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The  U.S.  Department  of  Defense  (DoD), 
which  has  seen  a  decade  of  growth  in  expendi¬ 
tures,  now  spends  about  $400  billion  annually 
on  acquiring  systems  and  services,  in  roughly 
equal  measures  for  each.  The  next  decade  will 

likely  see  a  marked  reduction  in  defense  acquisition  spending,  as  DoD 
leadership  is  aiming  to  save  2-3  percent  annually  through  productivity 
enhancements  and  greater  efficiencies.  To  realize  these  savings,  DoD 
must  first  know  what  is  being  saved  and  how  to  measure  it.  But  this 
is  not  simply  a  question  of  establishing  some  numerical  target  and 
comparing  output  against  it.  As  Frank  Kendall,  Principal  Deputy  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  said  of 
a  recent  personnel  initiative,  "We  are  not  measuring  performance  in 
terms  of  pure  numbers. ..We  want  quality  and  we  want  the  right  kinds 
of  people”  (Brodsky,  2010,  p.  2). 

The  idea  of  adopting  broader  ways  of  assessing  defense  acquisi¬ 
tion  is  at  the  heart  of  this  issue’s  theme,  "Measuring  Programs  and 
Progress.”  Chiang  H.  Ren,  Col  Stephen  Busch,  USAF  (Ret.),  and  Mat¬ 
thew  Prebble  begin  their  discussion  by  arguing  for  a  "try-before-fly” 
approach  to  automated  information  system  programs,  by  creating  a 
soft  start  period  to  evaluate  materiel  solution  concepts  before  com¬ 
mitting  to  a  Materiel  Development  Decision.  Rick  S.  Thomas,  N.  Clark 
Capshaw,  and  Paul  M.  Franken  examine  how  a  spectrum  of  testing 
activities  was  used  to  assess,  over  a  period  of  time,  the  evolutionary 
progress  of  a  system-of-systems,  to  build  up  an  integrated  evaluation 
of  its  performance. 

Of  course,  evaluating  progress  requires  understanding  which  criti¬ 
cal  factors  must  be  evaluated,  as  Steve  Geary,  Scott  Koster,  Wesley  S. 
Randall,  and  Jeffrey  J.  Playnie  explain  in  their  article  on  Life  Cycle  Sup¬ 
port  Strategies.  Ismail  Cicek  and  Capt  Gary  S.  Beisner,  USAF,  describe 
how  they  were  able  to  save  program  time  and  money  by  exploring 
new  test  and  analyses  methods  to  assess  aeromedical  equipment  for 
safety  certification.  The  final  article  by  Alan  Haggerty  and  Roy  Wood 
describes  how  a  little-known  company  at  the  beginning  of  World  War 
II  (WWII)  quickly  developed  the  renowned  P-51  Mustang  by  pulling 
together  a  series  of  technological  developments  into  a  well-integrated 
system;  with  no  time  for  systematic  measurement  and  testing,  the 
engineers  relied  on  their  collective  lifetimes  of  experience  to  field  a 
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capability  that  ultimately  helped  win  the  war. 

By  any  measure,  today’s  acquisition  workforce  is  as  experienced 
and  as  motivated  as  their  forerunners  in  WWII.  Those  qualities  will  be 
needed  in  the  years  to  come  as  we  rise  to  the  challenge  of  “doing  more 
without  more”  in  equal  measure. 


Dr.  Larrie  D.  Ferreiro 
Executive  Editor 
Defense  ARJ 
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WE  ARE  CHANGING  OUR  NAME! 

Our  name  will  officially  change  from  the  Defense  Acquisition 
Review  Journal  to  the  Defense  Acquisition  Research  Journal  effective 
January  2011— Issue  57.  Although  the  acronym  ARJ  will  remain  the 
same,  the  name  change  is  being  implemented  to  reflect  an  overall 
commitment  to  refocus  the  Defense  Acquisition  University’s  research 
efforts  on  strategic  alignment  to  meet  the  requirements  of  the 
Defense  Acquisition  Workforce. 


Norene  Fagan-Blanch 
Managing  Editor 
Defense  ARJ 
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•  to  facilitate  learning  and  knowledge  sharing  in  conjunction  with  the  theme  of  the  annual  DAU  Acquisition 
Community  Symposium 

•  to  generate  acquisition-related  research  studies/arlides  for  the  Defense  Acquisition  Review  Journal  (ARJ I. 
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•  The  competition  is  open  to  anyone  interested  in  the  DoD  acquisition  system  and  is  not  limited  to  government  or 
contractor  personnel. 

•  Employees  of  the  federal  government  (including  military  personnel)  are  encouraged  to  compete  and  ore 
eligible  for  cash  awards  unless  the  paper  was  researched  or  written  as  part  of  the  employee's  official  duties  or 
was  done  on  government  time.  If  the  research  effort  is  performed  os  part  of  official  duties  or  on  government 
time,  the  employee  is  eligible  for  o  non-cash  prize,  i.e.,  certificate  and  donation  of  cosh  prize  to  a 
CFC-registered  charity  of  winner's  choice. 

•  First  prize  is  SI, 000.  Second  prize  is  $500. 

•  The  format  of  the  paper  must  be  in  accordance  with  guidelines  for  articles  submitted  for  the  Defense  ARJ, 
which  can  be  found  on  the  DAU  Web  site  at  http-y/www.dau.mil/pubscals/Pages/ARJ.ospx. 

•  The  theme  for  201 1  is:  "Making  Every  Dollar  Count  —  Improving  Acquisition  Outcomes." 

•  Research  papers  are  due  November  1, 2010. 

•  Popers  should  be  submitted  to  the  DAU  Director  of  Research:  Dr.  Larrie  D.  Ferreiro,  703-805-5423  or 
larrie.ferreiro@dau.mil. 

•  Papers  will  be  evaluated  by  a  panel  selected  by  the  DAUAA 
Board  of  Directors  and  the  DAU  Director  of  Research. 

•  Winners  will  be  announced  and  papers  will  be  presented 
at  the  DAU  Acquisition  Community  Symposium  in  April  2011. 


Submissions  are  due  by  November  1, 2010. 

For  more  information  visit  www.dauaa.org. 
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INFORMATION  SYSTEMS 

{  Chiang  H.  Ren,  Col  Stephen  Busch,  USAF  (Ret.), 
and  Matthew  Prebble 

The  success  rate  for  acquiring  automated  information  systems 
(AIS)  continues  to  be  a  source  of  considerable  interest  to 
Congress  and  the  Department  of  Defense.  Building  upon 
Defense  Science  Board  recommendations,  the  authors’  research 
identifies  improvements  for  initiating  information  systems 
acquisition.  Adding  a  soft  start  period  prior  to  the  Materiel 
Development  Decision  allows  the  acquisition  community  to 
negotiate  with  end  users  regarding  system  concepts  that  can 
satisfy  the  materiel  solution  concept  and  better  manage  the 
flexibility  of  AIS  concepts  to  lower  risks.  This  management 
is  enabled  through  a  newly  formulated  reference  frame  that 
maps  the  materiel  solution  concept  to  the  system  concept  and 
allows  the  system  concept  to  be  reorganized  and  optimized 
prior  to  the  analysis  of  acquisition  approaches. 
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The  U.S.  Congress  established  new  reporting  requirements  and 
performance  constraints  for  Major  Automated  Information  System 
(MAIS)  acquisition  programs  in  section  816  of  the  FY  07  National 
Defense  Authorization  Act  (NDAA,  2006).  These  requirements, 
codified  in  10  United  States  Code  (U.S.C.),  chapter  144A,  were  later 
modified  to  include  pre-MAIS  programs  and  a  completion  standard 
of  5  years  from  first  obligation  of  funds  to  full  deployment  decision. 


Defense  Science  Board  and  Acquisition  of 
Information  Technology 

At  the  request  of  Congress,  the  Defense  Science  Board  (DSB) 
then  studied  the  DoD  policies  and  procedures  for  the  acquisition  of 
information  technology  (IT)  and  recommended  in  their  March  2009 
report  a  new  acquisition  process  for  IT  systems  based  on  commercial 
worldwide  best  practices  (Department  of  Defense  [DoD],  2009a). 
This  more  streamlined  process,  primarily  for  stand-alone  software 
development,  progressively  refines  the  software  product  based  on 
continuous  stakeholder  participation  and  multiple  iterations  leading 
to  a  major  release.  The  DSB  task  force  further  recognized  that  the 
current  acquisition  process  for  all  DoD  systems,  as  presented  in  the 
December  8,  2008,  release  of  Department  of  Defense  Instruction 
(DoDI)  5000.02,  is  still  valid  for  IT  systems  acquisition,  with  substan¬ 
tial  trade-offs  in  design,  development  of  nonsoftware  technologies, 
and  integration  with  major  weapon  systems  (DoD,  2008).  The  Con¬ 
gressional  reporting  process  for  MAIS  programs  and  the  current,  as 
well  as  proposed,  acquisition  processes  for  satisfying  congressional 
expectations  are  presented  in  Figure  1. 

In  Section  804  of  the  FY  10  NDAA,  Congress  officially  called  for 
the  Secretary  of  Defense  to  develop  and  implement  a  new  acquisi¬ 
tion  process  for  IT  systems  based  on  the  DSB  recommendations 
(NDAA,  2009).  Regarding  the  selection  of  which  acquisition  process 
to  adopt  for  a  specific  IT  program,  the  DSB  members  in  their  report 
remarked,  “One  could  argue  that  if  the  leadership  and  program  man¬ 
agers  cannot  sort  out  this  high-level  decision,  they  have  no  chance 
of  effectively  managing  or  overseeing  the  programs.” 

Our  research  presupposes  that  this  DSB  remark  actually  strikes 
at  the  heart  of  the  problem  as  to  why  so  many  IT  programs  have 
faced  developmental  problems  in  recent  years.  Identifying  which  IT 
programs  involve  nonsoftware  technologies  and  embedding  software 
into  weapon  systems,  as  governed  by  interoperability  standards,  are 
relatively  straightforward.  But,  understanding  which  IT  programs 
do  not  involve  substantial  trade-offs  in  design  and  approach  can  be 
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FIGURE  1.  MAIS  REPORTING,  DoDI  5000.02,  AND  DSB 
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Note.  CDR  =  Critical  Design  Review;  DT  =  Developmental  Testing;  FOC  =  Full  Operational 
Capability;  IOC  =  Initial  Operational  Capability;  OT  =  Operational  Testing;  PDR  = 
Preliminary  Design  Review;  *  =  Iteration 


overtly  or  even  covertly  complex.  Much  of  this  complexity  starts  with 
how  one  defines  the  IT  system  to  be  developed. 

The  IT  system  can  be  one  small  application,  a  suite  of  applications, 
or  a  total  solution  for  the  enterprise  because  of  the  integrative  nature 
of  modern  IT  capabilities.  Over  the  past  decade,  overly  ambitious  IT 
programs  involving  multiple  integrated  or  flexible  components  have 
ultimately:  (a)  decoupled  into  independent  development  paths,  (b) 
collapsed  due  to  escalating  baselines,  (c)  breached  schedules  and 
costs,  and/or  (d)  failed  to  fully  meet  user  expectations.  At  the  same 
time,  dozens  of  small  IT  programs  have  been  initiated  across  the 
DoD  with:  (a)  redundancies  in  development  activities,  (b)  overlaps  in 
functionality,  (c)  barriers  toward  integration  or  federation,  and/or  (d) 
limitations  in  scalability.  Defining  the  IT  system  concept  correctly  at 
the  initiation  of  acquisition  activities  is  therefore  critical  to  both  the 
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decision  to  use  a  streamlined  IT  acquisition  process  and  the  approach 
for  leveraging  the  rigors  of  the  current  DoDI  5000.02  process  to 
manage  developmental  risks.  This  article  will  explore  methods  for 
improving  the  initiation  of  acquisition  activities  through  better  con¬ 
cept  definition  of  the  IT  system. 


Analysis  Methodology 

The  objective  of  this  research  endeavor  is  to  determine  acquisi¬ 
tion  process  improvements  for  future  automated  information  systems 
without  dwelling  on  the  program  deficiencies  and  failures  of  the  past. 
Therefore,  a  methodology  of  process  decomposition  and  functional 
advancement  is  adopted  in  lieu  of  case  studies.  The  first  step  in 
decomposition  is  to  identify  the  point  of  disconnect  between  the  IT 
materiel  solution  concept  and  IT  system  concept  formulation  within 
the  current  process.  This  point  then  permits  the  insertion  of  a  trans¬ 
formation  function,  which  maps  the  user  materiel  solution  and  the 
associated  system  concept  to  a  common  reference  frame.  This  refer¬ 
ence  frame  then  enables  the  reorganization  of  the  system  concept 
for  more  effective  acquisition  while  preserving  connectivity  to  user 
requirements.  An  inductive  analysis  of  current  terminologies  in  the  IT 
community  is  used  to  create  the  reference,  and  a  deductive  analysis 
of  the  relationships  within  the  reference  frame  is  used  to  establish 
the  application  methodology.  Finally,  recommended  changes  to  the 
acquisition  process  are  formulated  using  the  analytical  results. 


Analysis  Results 

Currently,  the  initiation  of  acquisition  activities  for  major  IT  sys¬ 
tems  within  the  DoD  is  based  on  a  Materiel  Development  Decision 
(MDD)  by  the  Milestone  Decision  Authority  after  consideration  of 
requirements  generated  through  the  Joint  Capabilities  Integration 
and  Development  System  (JCIDS)  and  the  acquisition  community's 
ability  to  satisfy  the  requirements  (Joint  Chiefs  of  Staff,  2007).  The 
JCIDS  process  of  capabilities-based  assessments,  functional  area 
analysis,  and  functional  needs  analysis  ensures  that  the  established 
need  for  a  materiel  solution  (physical  system)  results  in  an  essen¬ 
tial  capability  for  the  warfighter.  The  resulting  Initial  Capabilities 
Document  must  accurately  address  operational  gaps,  align  with  the 
integrated  operational  concepts  of  the  regional  and  functional  com¬ 
batant  commanders,  and  present  validated  requirements. 
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The  analysis  of  the  acquisition  community's  ability  to  satisfy  the 
requirements  must  take  into  account  technologies  from  all  sources 
and  explore  solution  options  in  the  context  of  the  DoD  Architec¬ 
tural  Framework  (Wisnosky  &  Vogel,  2004).  The  DoD  Architectural 
Framework  used  in  capabilities-based  analysis  starts  with  a  high-level 
operational  concept  graphic  as  the  first  operational  view  and  then 
quickly  progresses  down  levels  of  detail  in  operational  understand¬ 
ing,  systems  definition  and  configuration,  and  technical  standards. 

The  current  acquisition  initiation  process,  as  presented,  extends 
from  a  tradition  of  preserving  the  purity  of  user/warfighter  needs, 
with  the  acquisition  community  responding  to  these  needs.  The 
concept  of  evolutionary  acquisition  allows  the  acquisition  commu¬ 
nity  response  to  evolve  in  increments  driven  by  stages  in  technology 
maturation.  However,  the  assumption  is  still  that  the  user-defined 
materiel  solution  concept  can  easily  extend  to  a  system  concept  for 
acquisition.  This  assumption  remains  quite  valid  for  mechanical  sys¬ 
tems.  Obviously,  it  would  be  ridiculous  if  the  warfighters  want  a  plane 
and  the  acquisition  community  tells  the  warfighters  that  what  they 
really  want  is  some  wings,  avionics,  and  jet  engines.  Even  when  the 
warfighters  want  a  system  of  systems,  such  as  missiles,  radars,  and 
control  centers,  the  nature  of  the  systems  is  very  clear. 

The  assumption  of  clear  system  concept  is  not  necessarily  true 
in  IT  acquisition.  For  example,  if  the  warfighters  want  an  integrated 
suite  of  applications  to  create  an  enterprise  solution  environment,  it 
would  not  be  silly  for  the  acquisition  community  to  tell  the  warfighters 
that  the  enterprise  can  be  better  supported  by  more  loosely  coupled 
applications  developed  as  multiple  programs.  Essentially,  when  users 
have  an  IT  solution  concept,  that  single  concept  could  equate  to  a  sin¬ 
gle  system,  multiple  systems,  a  system  of  systems,  or  even  a  mixture 
of  systems  and  services.  An  eagerness  of  the  acquisition  community 
to  rush  down  one  path  when  given  validated  requirements  can  lead 
to  great  program  risks.  The  complexity  of  the  initiation  process  can¬ 
not  be  ignored. 

A  hierarchical  architectural  framework  may  not  be  sufficient  to 
map  the  complex  understanding  of  requirements  into  a  paradigm 
where  effective  system  or  systems  definitions  can  be  formulated.  To 
create  a  complementary  framework,  we  examined  the  ways  in  which 
IT  is  described  in  industry  and  concluded  that  interlinked  dimensions 
of  characterization  can  be  used  to  better  define  IT  systems.  These 
dimensions  are  not  clearly  elucidated  in  literature,  in  part  due  to  the 
competing  approaches  and  schools  of  thought  in  commercial  soft¬ 
ware  development.  As  a  result,  a  better  standardized  and  organized 
set  of  parameters  for  capturing  the  dimensions  of  DoD  IT  system 
characterization  is  still  merited. 
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FIGURE  2.  REFERENCE  FRAME  FOR  MAPPING  MATERIEL 
SOLUTION  CONCEPT  TO  IT  SYSTEM  CONCEPT 
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Our  research  suggests  that  once  a  materiel  solution  concept 
with  operational  requirements  has  been  formulated  by  the  users,  the 
acquisition  community  can  place  that  concept  into  a  three-dimen¬ 
sional  reference  frame  as  shown  in  Figure  2.  Based  on  initial  market 
research  and  technology  assessment  prior  to  MDD,  the  solution  con¬ 
cept  can  be  mapped  across  a  range  of  blocks  within  the  reference 
frame  and  be  defined  through  the  identification  of  activities  in  each 
block.  The  scalar  parameters  for  establishing  each  of  the  three  dimen¬ 
sions  are  proposed  and  explained  in  the  discussion  that  follows.  The 
methods  of  defining  IT  systems  using  this  mapping  of  the  solution 
concept  are  also  presented. 

Dimension  1:  Mapping  the  Magnitude  Level  of  the  Concept 

Software/computer  codes,  unlike  mechanical  devices,  have  a 
greater  ability  to  be  both  decomposed  into  progressively  smaller 
individual  units  and  integrated  into  progressively  larger  overarching 
units.  Therefore,  defining  a  system  based  on  bounding  software  in 
the  modern  era  of  distributed  computing  is  almost  as  hard  as  find¬ 
ing  boundaries  within  a  continuous  body  of  water.  The  current  state 
of  software  technology  does  suggest  that  four  levels  of  boundaries 
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can  be  established.  These  boundaries  are  four  parameters  on  a  scale 
that  indicate  the  levels  and  total  magnitude  of  software  organization. 
When  mapping  a  solution  concept  to  this  scale,  software  organization 
can  reach  the  point  of  being  an  application  (Level  3)  or  creating  an 
environment  (Level  4)  without  first  having  an  extensive  number  of 
modules  (Level  2)  or  subroutines  (Level  1). 

Also,  software  modules  and  subroutines  can  sometimes  become 
stand-alone  applications,  and  software  applications  can  sometimes  be 
recategorized  as  being  able  to  establish  an  environment.  The  distinc¬ 
tiveness  of  these  four  parametric  levels,  therefore,  requires  integration 
with  the  other  two  dimensions— Scope  and  Source.  Otherwise,  the 
description  of  activities  at  each  level  can  become  inconsistent  from 
one  acquisition  effort  to  the  next. 


LEVEL  1  (SUBROUTINES) 

Portions  of  code  that  perform  specific  tasks  or  functions  within 
the  context  of  an  overarching  program/executable  file.  Subroutines 
can  be  reused,  and  popular  subroutines  can  be  stored  as  a  task  library 
for  easy  access.  In  a  program,  subroutines  can  be  within  subroutines 
to  form  a  nested  structure  and/or  be  integrated  through  commands 
and  shared  parameters  (Fischer,  2001,  pp.  1-8). 


LEVEL  2  (MODULES) 

Independently  executable  files/programs  that  perform  func¬ 
tions  capable  of  being  organized  and  integrated  through  interfaces 
to  satisfy  the  design  capabilities  of  an  application.  Modules  built  on 
the  principles  of  component-based  engineering  can  be  reused  and 
rearranged  to  achieve  multiple  capabilities  (Szyperski,  2002).  Also, 
modules  of  foundational  utility  can  be  offered  as  a  service  through 
Service-Oriented  Architectures  (Bell,  2008). 


LEVEL  3  (APPLICATIONS) 

IT  products  that  meet  the  performance  of  defined  capabilities. 
Applications  will  generally  have  a  functional  architecture,  and  the 
architecture  can  be  designed  to  be  open  to  the  reorganization,  updat¬ 
ing,  and  upgrading  of  constituent  modules/components. 


LEVEL  4  (ENVIRONMENTS) 

IT  infrastructures  consisting  of  a  network  of  applications  including 
the  core  applications  that  sustain  the  environments.  Environments  are 
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generally  designed  to  satisfy  the  total  mission  needs  of  the  entity  they 
support.  And,  environments  can  be  network-centric,  highly  restricted 
and  compartmentalized,  or  centralized. 

Dimension  2:  Mapping  the  Utility  Scope  of  the  Concept 

Once  a  concept  has  been  mapped  to  levels  of  software  organiza¬ 
tion,  each  level  of  the  software  can  have  different  scopes  in  utility  as 
defined  in  the  following  discussion.  The  most  straightforward  align¬ 
ment  is  that:  (a)  a  software  environment  can  cover  the  entire  society 
impacted  by  a  DoD  mission,  (b)  a  software  application  within  the 
environment  can  support  a  Service  or  joint  enterprise,  (c)  a  software 
module  can  address  the  functional  needs  of  one  or  more  organiza¬ 
tions  within  the  enterprise,  and  (d)  a  software  subroutine  can  execute 
a  specialized  task  for  a  specific  unit  of  the  organization.  However,  the 
scope  of  utility  does  not  have  to  match  the  level  of  software  orga¬ 
nization  in  a  parallel  manner.  A  software  environment  or  application 
can  serve  only  the  mission  of  a  specific  unit.  A  software  subroutine 
or  module  can  alternatively  serve  the  mission  of  an  entire  enterprise 
or  society. 

A  potential  mistake  in  acquisition  planning  is  overlooking  the 
fact  that  the  subroutines  and  modules  used  to  create  applications 
and  environments  may  either  have  a  more  individually  unique  scope 
or  a  broader  scope  than  the  application  to  which  they  belong.  An 
individually  unique  scope  implies  the  need  for  greater  fidelity  in 
stakeholder  participation,  while  a  broader  scope  implies  latent 
potential  for  DoD  reuse  or  co-development.  To  better  understand 
the  varying  scopes  associated  with  the  elements  at  each  level, 
the  relationship  of  levels  and  scopes  can  be  captured  in  a  four-by- 
four  matrix. 


SCOPE  1  (UNIT  SUPPORT) 

Tasks,  functions,  and/or  capabilities  are  designed  around  sup¬ 
porting  specific  divisions  within  DoD.  A  high  degree  of  specificity, 
characteristic  of  Scope  1,  is  tailored  to  the  unique  needs  of  small  user 
groups. 


SCOPE  2  (ORGANIZATIONAL  SUPPORT) 

Tasks,  functions,  and/or  capabilities  are  designed  around  sup¬ 
porting  major  commands  and  agencies  within  DoD.  Scope  2  places 
emphasis  on  meeting  needs  associated  with  the  specific  mission  of 
the  organization. 
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SCOPE  3  (ENTERPRISE  SUPPORT) 

Tasks,  functions,  and/or  capabilities  are  designed  around  sup¬ 
porting  the  entire  DoD  community  or  a  military  service  within  the 
DoD.  Scope  3  places  emphasis  on  bringing  the  community  into  using 
a  single  conformance  standard. 


SCOPE  4  (SOCIETAL  SUPPORT) 

Tasks,  functions,  and/or  capabilities  are  designed  around  sup¬ 
porting  activities  and  awareness  among  all  societal  stakeholders  in  a 
DoD  mission.  Stakeholders  can  be  other  government  agencies,  state 
and  local  government  organizations,  international  organizations,  cor¬ 
porate  entities,  foreign  governments,  and/or  U.S.  as  well  as  foreign 
citizens  with  a  need  to  respond. 

Dimension  3:  Mapping  the  Delivery  Sources  for  the  Concept 

After  the  solution  concept  has  been  mapped  into  the  matrix  of 
levels  and  scopes,  each  box  in  the  matrix  can  then  have  a  unique 
source  for  providing  service  to  the  user.  As  with  scope,  the  source 
for  elements  at  each  level  can  be  from  one  to  all  four  of  the  following 
access  methods.  Further,  the  nature  of  access  at  higher  levels  may  be 
different  than  access  at  lower  levels.  Software  modules  and  subrou¬ 
tine  libraries,  for  example,  can  be  deployed  at  specific  user  sites  and 
from  user  servers.  However,  once  these  modules  or  subroutines  are 
integrated  with  other  modules,  the  total  resulting  application  can  be 
accessible  from  DoD  networks.  Alternatively,  software  modules  can 
be  acquired  as  a  cloud  computing-based  service  or  network-based 
tool.  Then,  these  modules  could  be  offered  to  users  as  a  part  of  local 
or  even  site-specific  applications.  Sometimes,  users  may  not  even 
be  aware  that  their  application,  with  personalized  features,  may  rely 
upon  capabilities  that  are  delivered  across  the  Internet. 


SOURCE  1  (EQUIPMENT-SPECIFIC) 

IT  system  element  resides  within  a  single  hardware  associated 
with  a  specific  user  or  group  of  users.  For  example,  software  loaded 
onto  user  desktops  and  portal  devices. 


SOURCE  2  (LOCALLY  DISTRIBUTED) 

IT  system  element  resides  within  multiple,  interconnected 
hardware  equipment  across  the  user  facility  (peer-to-peer  con¬ 
nectivity)  or  is  accessible  by  multiple  hardware  equipment 
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through  connection  to  central  hubs  (local  area  networks).  For 
example,  software  with  multiple  user  licenses  offered  through  a  local 
Intranet. 


SOURCE  3  (NETWORK-BASED) 

IT  system  element  hosted  at  a  DoD  recognized  facility  that  is 
accessible  across  an  entire  DoD  validated  network  (wide  area  net¬ 
works,  router  networks  across  the  Internet,  etc.).  For  example,  DoD 
developed  tools  offered  to  the  enterprise  through  DoD  portals. 


SOURCE  4  (CLOUD-BASED) 

IT  system  capability  accessible  as  a  scalable  service  across  the 
Internet  (Knorr  &  Grumman,  2008).  For  example,  commercial  tools 
from  multiple  redundant  sites  that  are  available  to  DoD  users  in  a 
device-  and  location-independent  manner. 


Application  of  the  Reference  Frame 

The  definition  of  the  three-dimensional  reference  frame  (Figure  2) 
permits  a  materiel  solution  concept  to  be  mapped/decomposed  into 
a  set  of  blocks  from  which  a  system  or  multiple  system  concepts  can 
be  formulated.  Figure  3  presents  a  notional  configuration  of  blocks 
that  represents  a  concept.  The  goal  in  describing  the  activities  in 
each  block  is  not  to  achieve  overwhelming  detail  but  to  understand 
the  relationship  of  blocks  to  one  another.  As  the  relations  of  activities 
across  the  blocks  are  all  achieved  through  codes,  the  correction  and 
optimization  of  relationships  through  systems  formation  are  feasible 
and  could  be  critical  to  acquisition  success.  Four  methods  of  systems 
analysis,  using  this  reference  frame,  can  support  the  organization  of 
acquisition  activities. 

Method  V.  Forming  Multiple  Systems  or  Reduced  Systems 
Based  on  Decomposition  of  the  Mapped  Concept 

The  relationships  between  blocks  from  a  mapped  concept  may 
have  natural  weak  points  where  forced  integration  yields  acquisi¬ 
tion  risks.  These  weak  points  may  merit  the  breakup  of  the  materiel 
solution  concept  into  separate  systems  for  acquisition.  Weak  points 
could  be  places  where  there  will  be:  (a)  high  risks  or  low  benefits 
for  integration  in  terms  of  schedule,  cost,  and  performance,  (b)  high 
stress  in  sustaining  integration  because  of  varying  pace  in  tech¬ 
nology  advancement,  and/or  (c)  extreme  challenges  in  achieving 
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FIGURE  3.  NOTIONAL  EXAMPLE  OF  BLOCK  CONFIGURATION 
MAPPED  FROM  SOLUTION  CONCEPT 
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integration  due  to  technology  scaling  and  new  technology  require¬ 
ments.  The  breakup  can  be  along  any  of  the  three  dimensions  and 
can  follow  a  complex  path  of  weak  points.  Some  attempts  at  creating 
an  integrated  software  environment  may  be  better  pursued  as  the 
development  of  separate  applications.  Some  attempts  at  trying  to 
provide  capability  to  the  society  or  enterprise  may  be  better  pursued 
by  providing  separate  capabilities  to  smaller  sets  of  key  organizations. 
In  addition,  some  attempts  at  delivering  services  through  the  Inter¬ 
net  may  be  better  pursued  as  a  delivery  of  more  controlled  services 
through  secure  networks  and  portals.  The  objective  is  an  end  state 
characterized  by  a  stable  configuration  of  blocks  to  create  system 
concepts  before  the  start  of  the  acquisition  process. 

Method  2:  Forming  a  Single  System  or  Reduced  Set  of  Systems 
Based  on  Integration  of  Multiple  Mapped  Concepts 

The  configuration  of  blocks  from  a  mapped  concept  may  lend 
itself  to  being  integrated  with  configurations  of  blocks  associated  with 
other  materiel  solution  concepts.  If  so,  an  opportunity  may  exist  to 
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develop  a  single  system  that  can  satisfy  multiple  sets  of  requirements. 
Factors  to  consider  when  studying  the  opportunity  for  integration 
include:  (a)  the  ability  to  consolidate  the  integrated  configuration  to 
reduce  schedule  and  cost,  (b)  the  ability  to  enhance  performance 
through  cross-leveraging  of  activities,  (c)  the  challenges  in  achiev¬ 
ing  integration,  and  (d)  the  risks  in  integration.  The  simplest  way  to 
integrate  is  to  identify  all  the  overlapping  block  definitions  along  the 
three  dimensions  and  create  the  hybrid  configuration  based  on  the 
overlaps.  Alternatively,  one  can  study  the  definitions  of  the  blocks  on 
all  sides  to  see  whether  a  redefined  set  of  blocks  can  satisfy  all  the 
configurations  to  be  integrated. 

Method  3:  Establishing  Co-Development  Activities 
Between  Multiple  Mapped  Concepts 

Some  blocks  in  a  mapped  concept  may  exhibit  common  char¬ 
acteristics  with  blocks  in  other  materiel  concepts  even  though  the 
separate  configurations  will  not  be  integrated.  Such  commonalities 
suggest  that  co-development  or  technology  sharing  opportuni¬ 
ties  are  still  available.  Given  the  complexity  of  software  structures, 
commonalities  at  the  lower  levels,  scopes,  and  sources  may  not 
always  be  obvious  when  comparing  overarching  concepts.  Therefore, 
comparing  blocks  along  each  of  the  three  dimensions  is  important. 
A  commonality  of  software  elements  at  any  level  clearly  encour¬ 
ages  coordination  between  systems  acquisitions.  However,  even  a 
commonality  of  users  at  any  scope  suggests  a  coupling  of  funding  pri¬ 
orities.  Also,  a  commonality  of  delivery  mechanisms  from  any  source 
suggests  a  coupling  of  infrastructure  and  supporting  hardware. 

Method  4:  Forming  New  Systems  Based  on  Reorganization  or 
Redefinition  of  Activities  for  Mapped  Concepts 

The  most  optimal  sequencing  and  timing  of  developmental  activi¬ 
ties  for  the  blocks  may  not  be  that  suggested  by  the  initially  mapped 
concept.  Optimal  timing  may  require  grouping  the  blocks  into  evo¬ 
lutionary  increments  or  iterations  within  one  major  release.  Optimal 
sequencing  may  require  some  blocks  to  be  redefined  and  others  to 
be  eliminated  to  facilitate  the  efficiency  of  acquisition. 

The  ability  to  organize  IT  acquisition  activities  into  all  manner 
of  partial  product  releases  under  the  spiral  development  concept 
is  a  two-edged  sword.  When  done  effectively,  the  timing  of  partial 
releases  can  take  advantage  of  technology  maturation  stages,  avail¬ 
ability  of  supporting  commercial  products,  user  feedback  for  product 
refinement,  and  early  mission  support  opportunities.  However,  iden¬ 
tifying  the  right  pieces  for  release,  timing  of  release,  and  extent  of 
release  can  be  difficult.  Too  many  releases  may  eat  away  the  time 
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periods  for  refining  products.  Too  few  releases  may  lead  to  greater 
adjustment  of  developmental  activities.  Further,  poorly  timed  releases 
may  result  in  less  beneficial  user  feedback,  missed  opportunity  of 
mission  impact,  and  wasting  of  test  and  certification  resources. 

Understanding  the  natural  breakpoints  in  a  system  concept 
could  reduce  the  level  of  error  in  sequencing  acquisition  activities 
and  determine  conditions  for  partial  product  release.  The  proposed 
reference  frame  could  aid  in  this  understanding  throughout  the 
acquisition  process. 


Conclusions 

The  systems  acquisition  process  in  DoDI  5000.02  and  the  tailored 
IT  systems  acquisition  process  proposed  by  the  DSB  both  have  a  hard 
starting  point  in  the  form  of  the  MDD.  Our  research  suggests  that  the 
IT  systems  acquisition  process  can  benefit  from  a  soft  start  period 
where  the  acquisition  community  can  negotiate  with  the  user  commu¬ 
nity  regarding  system  concepts  that  can  satisfy  the  materiel  solution 
concept  resulting  from  JCIDS.  This  period  also  allows  the  proposed 
acquisition  to  be  considered  within  the  context  of  all  acquisitions, 
paralleling  the  user  community  effort  to  ensure  that  the  materiel 
solution  concepts  fit  within  the  total  concept  of  joint  warfighting.  An 
MDD  Review  can  then  be  held  after  the  system  concept  has  been 
appropriately  established. 

The  risk  in  committing  to  materiel  development  without  an  accu¬ 
rately  defined  system  concept  is  that  the  acquisition  process  can 
quickly  advance  to  focusing  on  the  user  materiel  solution  concept 
as  the  default  path  to  a  system  concept.  Although  the  Analysis 
of  Alternatives  process  could,  and  maybe  should,  question  the 
materiel  solution  concept,  the  analyses  in  many  cases  have  been 
directed  toward  comparing  different  acquisition  approaches  instead 
of  questioning  what  is  to  be  acquired.  Even  with  the  alternatives  of 
“maintaining  the  status  quo  capability”  or  “do  nothing  about  achiev¬ 
ing  capability,”  the  requirements  and  associated  materiel  concept 
are  still  typically  used  as  the  measurement  baseline.  If  the  concept 
of  what  is  to  be  acquired  has  not  been  optimized  for  efficient  system 
acquisition  by  the  MDD  point,  the  resulting  challenges  could  continue 
uncorrected  throughout  the  acquisition  life  cycle. 

The  drive  to  satisfy  user  requirements  may  push  the  acquisi¬ 
tion  community  along  paths  of  overwhelming  integration,  overly 
ambitious  expectations  for  use,  and  overestimation  of  commercial 
capabilities.  When  the  complexity  of  acquisition  activities  exceeds 
the  government’s  ability  to  understand,  some  program  offices  may 
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FIGURE  4.  UTILITY  OF  THE  SOFT  START  PHASE  FOR  IT  SYSTEMS 
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attempt  to  acquire  the  capability  as  a  service,  anticipating  that  com¬ 
mercial  entities  will  assume  the  risks  of  development.  The  shifting 
of  risks  through  a  Service  Level  Agreement  (SLA)  does  not  elimi¬ 
nate  the  risks.  If  commercial  entities  cannot  master  the  nature  of 
government  system  needs,  then  all  an  SLA  can  do  is  to  reduce  the 
government’s  financial  risks  through  nonpayment  for  an  undelivered 
service.  However,  the  lack  of  service  capability  to  support  missions 
will  remain  a  problem. 

Soft  starts  in  the  acquisition  process  are  already  occurring  to 
reduce  technology  risks.  In  many  cases,  technology  projects  such 
as  Joint  Capability  Technology  Demonstrators  are  used  to  accel¬ 
erate  program  execution  to  capture  technology  opportunities  or 
delay  program  initiation  to  manage  technology  risks  (Department 
of  Defense,  2009b).  The  IT  soft  start  period  proposed  through  this 
research  (Figure  4)  will  require  far  less  resources  than  technology 
projects.  Further,  the  resources  should  come  from  a  general  pool  of 
existing  funds  to:  (a)  permit  early  acquisition  community  involvement, 
(b)  allow  integrated  examination  of  concepts  across  multiple  materiel 
solution  needs,  and  (c)  delay  the  first  obligation  of  funds  for  a  specific 
system  acquisition  until  after  MDD.  Through  the  soft  start,  the  risks  in 
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achieving  a  full-deployment  decision  5  years  after  the  first  obligation 
of  funds,  as  measured  by  Congress,  is  dramatically  reduced. 

In  DoDI  5000.02  (DoD,  2008),  the  Concept  Refinement  Phase 
was  redefined  and  renamed  the  Materiel  Solution  Analysis  Phase.  This 
change  may  more  accurately  reflect  the  activities  of  the  acquisition 
community  after  MDD.  However,  the  notion  of  concept  refinement 
may  still  be  valid.  Our  proposed  soft  start  period  can  be  named  the 
IT  System  Concept  Refinement  Phase,  which  leads  to  a  development 
decision  and  the  analysis  of  the  adopted  concept.  Once  IT  acquisition 
activities  have  been  more  accurately  initiated,  perhaps  the  debate 
regarding  which  tailored  acquisition  process  is  best  suited  for  a  spe¬ 
cific  system  acquisition  will  not  be  as  critical  because  all  processes 
can  be  more  easily  refined. 
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A  FRAMEWORK  FOR 
SYSTEM  OF  SYSTEMS 
EVALUATION 

WITHIN  AN  AIRBORNE 
INTELLIGENCE, 
SURVEILLANCE,  AND 
RECONNAISSANCE 
ENVIRONMENT 

i  Rick  S.  Thomas,  N.  Clark  Capshaw,  and 
Paul  M.  Franken 

Federal  test  and  evaluation  agencies,  particularly  those  asso¬ 
ciated  with  the  U.S.  military,  are  grappling  with  the  challenge 
of  evaluating  system  of  systems  ( SoS )  or  a  family  of  systems 
(FoS)— in  short,  developing  methods  whereby  the  contribu¬ 
tion  of  individual  systems  can  be  evaluated  when  operating  in 
combination  with  other  systems,  and  determining  the  effective¬ 
ness  when  various  subcomponents  are  added  or  removed  from 
the  overall  SoS.  In  this  article,  the  authors  present  a  proposed 
framework  for  conducting  such  evaluations  through  integrating 
developmental  testing,  operational  testing,  and  operational 
performance  data  into  the  evaluations.  A  recent  example  of 
the  evaluation  of  a  suite  of  aerial  intelligence,  surveillance,  and 
reconnaissance  (ISR)  systems  is  also  discussed,  relating  the 
aerial  ISR  evaluation  to  the  proposed  framework. 

Keywords:  System  of  Systems  (SoS);  Family  of 
Systems  (FoS);  Test  and  Evaluation;  Developmental 
Testing  (DT);  Operational  Testing  (OT);  Intelligence, 

Surveillance,  and  Reconnaissance  (ISR) 
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Federal  test  and  evaluation  agencies,  particularly  those  asso¬ 
ciated  with  the  U.S.  military,  are  grappling  with  the  challenge  of 
evaluating  system  of  systems  (SoS)  or  a  family  of  systems  (FoS)— in 
short,  developing  methods  by  which  the  contribution  of  individual 
systems  can  be  evaluated  when  operating  in  combination  with  other 
systems,  and  determining  the  effectiveness  when  various  subcom¬ 
ponents  are  added  or  removed  from  the  overall  SoS. 

This  is  particularly  challenging  when  trying  to  assess  airborne 
intelligence,  surveillance,  and  reconnaissance  (ISR)  sensors  employed 
as  an  SoS,  due  not  only  to  the  abundance  of  sensors  (Imagery  Intel¬ 
ligence  [IMINT],  Signals  Intelligence  [SIGINT],  Measurement  and 
Signals  Intelligence  [MASINT],  etc.),  but  also  the  myriad  ways  in  which 
these  sensors  can  be  used  in  combination  with  one  another  to  achieve 
mission  effects.  Further  complicating  such  an  evaluation  is  the  notion 
that  ISR  sensors  are  employed  at  different  levels,  both  in  the  sense 
of  mission  focus— national,  strategic,  operational,  and  tactical  (N,  S, 
O,  T)— and  physical  altitude. 

The  aim  of  this  article  is  to  develop  and  present  a  framework  for 
how  an  SoS  evaluation  might  be  realized.  It  begins  by  noting  the  spe¬ 
cial  challenges  of  evaluating  an  SoS,  introduces  an  analogy  to  aid  in 
further  discussion,  and  concludes  by  relating  this  analogy  to  a  recent 
U.S.  Army  Test  and  Evaluation  Command  (ATEC)  SoS  evaluation  of  a 
suite  of  aerial  ISR  systems.  The  article  also  relates  this  framework  to 
other  recently  published  evaluation  frameworks. 

In  the  examples  discussed  in  this  article,  ATEC  used  a  systems- 
level  methodology  that  concentrated  on  an  effects-based  approach 
of  layers  of  sensors  distributed  across  multiple  aircraft.  In  focusing 
on  multiple  individual  systems,  ATEC  focused  on  the  effects  of  the 
systems  and  not  the  overall  element  or  family;  accordingly,  this  article 
will  reference  SoS  versus  FoS. 


A  Conceptual  Model  for  an  Airborne  ISR  SoS 

Consider  the  model  for  airborne  ISR  assets  presented  in  the  Table. 
Note  that  each  level  presents  different  mission  sets  and  capability 
gaps  that  must  be  addressed. 

As  illustrated  in  the  Table,  not  only  are  there  many  kinds  of  air¬ 
borne  ISR  sensor  types  (IMINT,  SIGINT,  and  MASINT,  etc.),  but  also 
many  different  mission  sets,  instantiations  (satellites,  manned  air¬ 
craft,  unmanned  aerial  vehicles),  and  levels  of  focus  characterize  the 
individual  sensors.  The  Table  is  misleading  in  one  respect;  it  shows 
an  apparent  clear  demarcation  from  a  mission  set  at  one  level  (e.g., 
tactical)  to  the  next  (e.g.,  operational).  In  practice,  these  clear  lines 
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TABLE.  A  TAXONOMY  OF  AIRBORNE  ISR  SENSORS 


Level 

Focus 

Examples  of 
Mission  Sets 

Instantiation 

Sensor  Type 

National 

Very 

wide 

Monitor  nuclear 

weapons 

development 
and  testing 

Satellites 

IMINT  still 

photography, 

experimental/ 

advanced 

and  esoteric 

sensors 

Strategic 

Wide 

Monitor  troop 
massing  at  border 
areas  for  potential 

attack 

High-altitude 

aircraft 

IMINT,  SIGINT, 

MASINT 

Operational 

Medium 

Monitor  troop 

movements  in  an 

existing  theater 
of  operations; 

monitor  activities  of 

personnel  associated 
with  enemy  networks 

Medium-to-low 

altitude  aircraft 

and  unmanned 

aerial  vehicles 

(UAVs) 

IMINT-wide 

area  focus, 

full-motion 

video,  aerial 
photography 

Tactical 

Narrow 

Observe  specific 
enemy  attacks 
and  activities, 
such  as  emplacement 
of  improvised 
explosive  devices, 
ambushes,  etc. 

Low-altitude 

aircraft  and 

UAVs 

IMINT  full 

motion  video 

are  at  best  fuzzy,  and  often  nonexistent.  Increasingly,  the  national 
level  wants  to  know  every  bit  of  intelligence  gained,  even  down  to 
the  tactical  level;  in  turn,  tactical  operations  incorporate  as  much 
national-level  intelligence  as  possible.  Trying  to  determine  what 
asset  contributes  most  effectively  and  its  position  within  the  Table 
is  a  tough  problem— one  which  a  rigorous  SoS  evaluation  should 
address— but  how? 


Evaluating  the  Sum  of  the  Parts 
or  the  Capability  of  the  Whole? 


By  definition,  an  SoS  is  an  amalgam  of  individual  systems,  each 
of  which  is  designed  to  perform  a  specific  function.  When  individual 
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systems  are  combined  into  a  greater  whole,  this  can  conceivably 
change  their  character  and  function. 

Testers  often  find  it  much  easier  to  conduct  evaluations  of  indi¬ 
vidual  systems  because  the  parameters,  threats,  and  variables  that 
are  part  of  the  individual  system’s  tests  are  not  complicated  or  influ¬ 
enced  by  other  systems  that  could  either  augment  or  degrade  the 
individual  system’s  inherent  capability. 

In  contrast,  when  evaluating  an  SoS,  multiple  additional  chal¬ 
lenges  surface  that  evaluators  need  to  consider.  These  challenges 
could  be  mitigated  during  an  individual  system  test,  so  long  as  the 
evaluation  teams  are  aware  of  force  structuring  of  the  SoS  before 
conducting  the  individual  test.  The  discussion  that  follows  addresses 
some  of  the  considerations  that  increase  the  complexity  of  effective 
SoS  testing. 

Optimization  of  Systems  When  Integrated  With  Complementary  Systems 

When  individual  systems  are  integrated  as  part  of  a  larger  SoS, 
an  evaluation  strategy  must  account  for  how  the  individual  systems 
work  to  the  complement  or  to  the  detriment  of  the  other  systems. 
To  effectively  evaluate  these  relationships,  systems  should  be  tested 
in  an  iterative  fashion,  first  evaluating  effectiveness  of  individual  sys¬ 
tems  through  isolated  tests,  and  then  determining  the  capability  of 
the  entire  SoS  while  looking  for  synergistic  benefits  that  may  pres¬ 
ent  themselves.  Compatibilities  can  be  determined  by  identifying  all 
of  the  potential  relationships  the  individual  systems  have  with  one 
another,  distinguishing  which  relationships  are  most  critical  to  the 
SoS,  weighting  the  importance  of  these  relationships,  and  then  build¬ 
ing  these  relationships  into  the  SoS  evaluation  strategy.  Duplicative 
capabilities  or  gaps  in  a  particular  capability  must  also  be  identified 
and  factored  into  the  overall  assessment. 

Force  Structure  for  an  SoS 

The  nature  of  SoS  employment  requires  that  certain  force  struc¬ 
ture  characteristics,  which  are  not  inherently  part  of  the  existing 
units,  be  in  place  to  synergize  the  SoS.  An  operational  unit  is  more 
readily  capable  of  integrating  an  individual  system  into  its  structure 
because  managing  the  capabilities  and  limitations  of  one  system  is 
a  skill  and  discipline  that  soldiers,  down  to  the  individual  level,  are 
trained  to  do.  However,  integrating  multiple  systems  with  multiple 
capabilities  and  limitations,  and  the  complex  relationships  between 
them,  is  significantly  more  challenging  for  a  soldier  in  an  operational 
environment  to  assimilate.  Additionally,  battle  command  training  and 
leader  development  for  the  employment  of  individual  systems,  as 
compared  to  an  SoS,  is  significantly  different.  With  SoS  employment, 
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all  stakeholders  must  be  trained  to  understand  the  often-complex 
relationships  that  exist  between  the  systems’  sensors  to  ensure  that 
each  system  sensor  is  employed  as  efficiently  and  effectively  as 
possible.  For  example,  a  specific  SoS  includes  assets  that  support  all 
echelons  of  battle  command,  yet  would  be  employed  at  the  tactical 
level,  thus  requiring  units  to  have  a  mechanism  in  place  to  maximize 
the  capabilities,  understand  the  echeloning  of  the  capability,  and 
synergize  the  entire  SoS. 

SoS  In-Theater  Evaluation  Versus  Domestic  Test  Range  Evaluation 

The  logistics  required  to  create  a  test  event  for  an  individual  sys¬ 
tem  are  relatively  simple  compared  to  the  logistics  for  an  SoS  test 
event.  Coordinating  the  presence  of  an  individual  system  at  a  test 
range  with  its  respective  personnel  is  a  feasible  task,  even  during  a 
time  of  war.  Conversely,  coordinating  the  presence  of  multiple  systems 
and  their  respective  personnel  at  a  domestic  test  range,  all  at  the 
same  time,  for  a  test  event  during  a  time  of  war  is  extremely  difficult, 
if  not  impossible.  Quite  often,  the  systems  that  comprise  the  SoS  have 
been  evaluated  individually  and  are  already  deployed  in  theater. 

The  foregoing  constraints  limit  SoS  evaluation  using  traditional, 
domestic  test  range  operational  test  and  evaluation  procedures,  thus 
making  in-theater  evaluation  a  more  attractive  alternative.  Yet,  when 
conducting  an  in-theater  evaluation,  additional  factors  must  be  con¬ 
sidered.  The  following  discussion  presents  two  factors  that  surfaced 
during  ATEC’s  most  recent  in-theater  SoS  evaluation. 


RED  FORCE  CONSIDERATIONS 

The  red  force  component  that  would  be  used  in  a  domestic 
operational  test  would  be  based  upon  the  intelligence  community’s 
best  assessment  of  the  tactics  and  capabilities  of  the  enemy.  Notwith¬ 
standing  the  best  efforts  of  the  intelligence  community,  assessments 
based  on  recent  intelligence  may  not  accurately  represent  activity  of 
the  enemy  at  that  particular  time.  An  in-theater  assessment,  however, 
measures  the  SoS  against  the  true  red  force,  and  thus  provides  the 
most  accurate  assessment  of  the  SoS  capabilities  and  limitations. 
Unfortunately,  knowledge  of  red  force  activities  in  an  actual  theater 
of  war  is  limited— a  situation  that  equates  to  SoS  evaluation  under 
uncertainty.  For  example,  consider  an  SoS  designed  to  find  enemy 
improvised  explosive  devices  (lEDs).  If  the  SoS  finds  five  lEDs,  did  it 
perform  well  or  poorly?  How  does  the  evaluator  know  how  many  lEDs 
were  emplaced  by  the  red  force?  Evaluators  can  count  the  number 
of  IED  explosions  that  were  documented  and  the  number  of  lEDs 
found  and  cleared,  but  they  have  no  idea  how  many  lEDs  remain 
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undiscovered  and  undetonated,  or  how  many  exploded  without 
documentation. 


CONCEPT  OF  OPERATIONS/TACTICS,  TECHNIQUES,  AND  PROCEDURES 
(CONOPS/TTP)  CONSIDERATIONS 

Blue  force  CONOPS  and  TTP  are  defined  for  individual  systems, 
but  are  often  informal  and  evolutionary  for  the  employment  of  the 
SoS.  This  presents  an  enormous  challenge  for  the  evaluator,  who  must 
develop  measures  of  effectiveness  for  the  SoS  based  upon  CONOPS 
and  TTP  that  are  still  under  development.  In  addition,  TTP  are  consis¬ 
tently  modified  based  on  enemy  response  to  blue  force  actions.  This 
constant  operational  dynamism  makes  it  difficult  to  formalize  mea¬ 
sures  that  are  relevant  to  the  environment  where  the  SoS  is  employed. 

Beyond  the  limitations  inherent  in  range  testing  or  in-theater 
evaluation,  the  hierarchical  relationship  that  exists  between  the 
employment  of  individual  systems  and  the  supporting  tasks  that 
enable  effective  operational  SoS-level  deployment  must  also  be 
considered  by  the  test  community  (Figure  1).  SoS  evaluation  should 
examine  the  entirety  of  factors  involved  in  SoS  employment,  applying 
a  multidisciplined  methodology  to  achieve  a  multidisciplined  measure 
of  operational  effect.  But,  doing  this  in  practice  can  quickly  become 


FIGURE  1.  SYSTEM  OF  SYSTEMS  PARADIGM 
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an  intractable  problem,  considering  the  number  of  variables  involved 
and  the  difficulty  in  altering  those  variables  one  at  a  time. 

The  capabilities  gained  through  modeling  and  simulation  (M&S) 
and  associated  software  packages,  focused  on  insurgency  opera¬ 
tions,  are  essential  to  the  overall  evaluation  of  an  SoS.  M&S  is  needed 
that  can  ingest  all  of  the  relevant  data  from  the  system,  assess  the 
environment  associated  with  insurgency  operations,  assess  the 
immediate  threat,  and  then  fuse  and  maximize  inputs  from  other 
systems  operating  under  the  same  parameters  to  identify  the  best 
mix  of  assets  at  the  strategic,  operational,  and  tactical  levels.  The 
results  from  SoS  evaluation  should  inform  the  community  of  not 
only  the  capability  of  the  enterprise  to  engage  and  be  successful, 
but  how  best  to  distribute  the  capabilities  for  the  greatest  overall 
operational  effect. 

The  following  hypothetical  example  taken  from  the  world  of 
sports  may  best  illustrate  the  concept  of  SoS  evaluation. 

An  Analogy  for  SoS  Evaluation 

Consider  an  analogy.  A  professional  football  coach  is  facing  a 
decision:  a  first-round  draft  pick  for  a  new  player.  This  opportunity  to 
draft  a  new  player  forces  the  coach  to  thoroughly  evaluate  the  array 
of  available  players  and  their  potential  contributions  to  the  team.  Does 
he  go  for  an  offensive  player  or  a  defensive  player?  Does  he  go  for 
a  quarterback,  a  lineman,  or  a  running  back?  If  he  chooses  a  quar¬ 
terback,  does  he  choose  one  from  a  college  that  has  an  aggressive 
passing  game,  or  one  from  a  school  that  has  a  run-based  offense? 

Now  consider  the  data  that  the  coach  has  for  making  his  decision. 
Undoubtedly,  he  has  accumulated  data  on  the  draft  choice’s  speed, 
weight,  strength,  and  performance  in  a  college  environment.  If  he  is 
thorough,  he  also  has  some  information  about  how  the  player  per¬ 
forms  in  a  high-stakes  environment  such  as  a  championship  game,  his 
injury  record,  and  how  he  fits  in  as  part  of  a  team.  Though  the  coach 
should  optimally  consider  individual  players’  skills  ( system  level )  in 
making  a  draft  decision,  his  decision  should  be  ultimately  based  on 
his  evaluation  of  the  player’s  potential  to  improve  the  overall  perfor¬ 
mance  of  the  team  (SoS  level). 

Finally,  the  coach  makes  his  decision  and  drafts  a  player.  But,  he 
doesn’t  put  the  new  player  in  as  a  starter  automatically.  Instead,  he 
uses  training  camp,  pre-season  games,  and  regular  season  substitu¬ 
tions  to  determine  whether  the  new  player  merits  a  starting  position 
with  the  team. 

Let’s  suppose  the  coach  has  done  all  of  this  throughout  the  new 
player’s  first  season,  and  at  the  end  of  the  season  the  team’s  record 
is  11-5— a  substantial  improvement  from  a  lackluster  8-8  perfor- 
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mance  the  previous  year.  Does  the  coach  attribute  the  entirety  of  the 
improvement  to  the  new  player?  Probably  not,  considering  the  team 
gained  a  number  of  new  players— some  from  the  draft,  others  from 
trades,  and  several  through  free  agency.  These  new  players,  as  well  as 
factors  beyond  the  coach’s  control  such  as  schedule  difficulty,  have 
affected  the  team’s  performance.  So,  how  does  the  coach  determine 
what  mixture  of  variables  contributed  to  the  team’s  improved  record? 
How  does  he  decide  which  players  to  retain  and  which  ones  to  shed 
for  the  next  season? 

Each  of  the  coach’s  inquiries  and  decisions  represents  a  stage  in 
the  evaluation  of  an  SoS,  and  it  is  instructive  to  the  development  of 
the  framework  to  now  extend  this  analogy  to  the  evaluation  of  an  SoS. 

Relationship  of  Sports  Analogy  to  an  Airborne  ISR  SoS  Evaluation 

The  sports  analogy  presented  earlier  illustrates  some  of  the  chal¬ 
lenges  of  conducting  an  airborne  ISR  SoS  valuation.  First  of  all,  are  the 
capability  gaps  clear?  Do  decision  makers  know  for  what  “positions” 
they  are  recruiting?  Is  there  a  gap  in  N,  S,  O,  T  mission  coverage,  or 
perhaps  all  four?  Is  there  a  reason  to  believe  that  an  airborne  ISR  asset 
can  fulfill  one  or  more  of  these  gaps? 

Second,  what  is  known  about  the  “players”— the  specific  airborne 
ISR  assets  that  might  be  added  to  an  SoS?  Much  of  this  type  of 
information  is  gleaned  from  developmental  testing  (DT).  DT  enables 
evaluators  to  gain  knowledge  of  the  technical  capabilities  of  the  ISR 
asset  (analogous  to  a  player’s  weight,  strength,  and  speed),  but  little 
or  no  knowledge  on  how  the  asset  will  perform  when  integrated  with 
the  “team.” 

Integration  into  the  “team”  begins  with  operational  testing  (or 
OT),  which  is  often  performed  in  an  artificial  environment,  such  as  one 
of  the  Army’s  proving  grounds.  Use  of  such  a  test  center  allows  evalu¬ 
ators  to  gain  a  better  understanding  of  how  the  system  undergoing 
testing  will  perform  when  used  in  an  operational  environment.  A  true 
operational  environment  will  also  enable  evaluation  of  the  system 
not  only  with  its  actual  “teammates”  (the  Blue  Force),  but  also  in  the 
presence  of  the  opposing  team  (the  Red  Force). 

Finally,  the  system  is  deployed  and  integrated  with  other  ISR 
assets,  all  of  which  are  dedicated  to  fulfilling  a  mission,  but  this  mis¬ 
sion  may  have  components  or  effects  at  each  of  the  N,  S,  O,  T  levels. 
Certain  mission-related  effects  are  measured,  and  things  seem  to  be 
improving;  coalition  causalities,  for  example,  have  declined. 

Now,  the  real  dilemma  begins!  How  is  it  possible  to  determine 
whether  one  particular  combination  of  assets  (a  “team”)  is  respon¬ 
sible  for  the  improvement,  or  whether  the  improvement  was  due 
to  exogenous  factors  (such  as  a  troop  surge  or  intelligence  from 
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an  enemy  defector)?  How  is  it  possible  to  determine  whether  cer¬ 
tain  assets  (the  “players”)  are  performing  well  at  their  positions,  or 
whether  it  would  be  better  if  certain  of  the  ISR  assets  were  “traded” 
for  other  assets? 


Example  of  a  Recent  ATEC  SoS  Evaluation 

ATEC  has  rich  experience  conducting  SoS  evaluations,  including 
the  recent  evaluation  of  an  Army  suite  of  aerial  ISR  systems  (an  SoS). 
Figure  2  also  shows  how  the  system ,  mission,  test,  and  evaluation  ele¬ 
ments  were  integrated  into  ATEC’s  recent  evaluation. 

A  general  discussion  to  demonstrate  how  the  framework  previ¬ 
ously  described  can  be  applied  to  similar  SoS  evaluations  follows. 

ATEC  began  its  SoS  evaluation  by  performing  thorough  DT  on 
each  individual  aerial  ISR  component  of  the  SoS.  Through  DT,  ATEC 
was  able  to  examine  the  technical  capabilities  of  each  of  the  sensors. 
In  this  case,  all  of  the  sensors  were  IMINT  sensors,  though  each  was 
employed  in  a  different  way  and  was  expected  to  add  a  unique  capa¬ 
bility  to  the  final  product. 

Then,  in  a  series  of  OT  scenarios  conducted  at  Yuma  Proving 
Ground,  ATEC  was  able  to  extend  the  technical  results  of  the  DT  to 
gain  a  better  understanding  of  these  aerial  ISR  systems  and  how  each 
might  perform  as  part  of  an  overall  SoS.  Although  data  were  limited 
due  to  the  operational  conditions  in  theater,  ATEC  used  scientific 
methods  to  complete  the  initial  study  of  the  SoS.  In  addition,  ATEC 
developed  test  and  threat  protocols  to  ensure  like  testing  across  a 


FIGURE  2.  ATEC  TFO  SoS  EVALUATION  FLOWCHART 
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series  of  systems,  and  then  supplemented  this  information  with  a 
forward  operational  assessment  (FOA)  team  that  collected  data  while 
systems  were  deployed  in  operational  conditions. 

ATEC’s  FOA  team  collected  performance  data  from  several 
sources:  end-of-mission  reports,  user  surveys,  commander  surveys, 
stakeholder  interviews,  white  papers,  and  reports  of  enemy  activities. 
Further,  ATEC  developed  and  used  the  Common  IED  Exploitation  Tar¬ 
get  Set  ontology  and  application  to  define  conditions  and  standards, 
and  model  results  for  OT  (Franken  et  al.,  2009). 

ATEC  extended  the  results  of  this  SoS  evaluation  by  using  mod¬ 
eling  and  analysis  (M&A)  methods.  M&A  methods  were  utilized  to 
explore  the  possible  mission  effects  of  employing  the  SoS  differently 
and  in  different  combinations  with  a  focus  on  insurgent  methods, 
timing,  and  location  of  attack.  The  use  of  M&A  created  a  virtual  test 
environment  for  this  SoS  evaluation,  and  helped  reveal  a  more  opera¬ 
tionally  effective  way  (i.e.,  system  composition,  user  tactics,  flight 
schedules,  etc.)  to  employ  this  SoS.  The  data  from  tests,  operational 
assessment,  and  threat  integration  were  used  to  conduct  first-order 
validation  of  the  insurgent  model.  More  detailed  verification  and  vali¬ 
dation  will  be  executed  as  data  become  available. 

Relationship  to  Other  Frameworks 

The  framework  presented  in  this  article  is  not  intended  to  be  used 
in  isolation,  but  in  combination  with  other  frameworks  that  can  help 
lend  clarity  to  the  evaluation  of  SoS.  Two  of  these  are  addressed  in 
the  following  discussion. 

Simmons  and  Wilcox  (2007)  introduced  the  notion  of  a  four- 
element  framework  for  test  and  evaluation,  encompassing  system, 
mission,  test,  and  evaluation  elements  in  an  integrated  whole. 

The  four-element  framework  provides  a  systematic  approach 
to  developing  a  T&E  plan  that  evaluates  mission  capabilities, 
system  effectiveness,  and  system  suitability.  The  mission  and 
system  elements  define  what  is  to  be  evaluated.  The  mission- 
to-system  interface  links  the  elements  together  and  ensures 
that  the  development  of  the  evaluation  and  test  elements 
always  remain  focused  on  the  unit’s  ability  to  execute  the  mis¬ 
sion  when  using  the  system.  This  provides  a  defined  guideline 
for  developing  the  evaluation  measures  and  a  roadmap  for 
how  the  tests  support  the  evaluation,  (p.  66) 

A  National  Research  Council  (2006)  report  cited  “continuous  pro¬ 
cess”  as  the  framework  upon  which  to  meet  the  challenge  of  testing 
in  an  evolutionary  acquisition  environment. 
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In  evolutionary  acquisition,  the  entire  spectrum  of  testing 
activities  should  be  viewed  as  a  continuous  process  of  gath¬ 
ering,  analyzing,  and  combining  information  in  order  to  make 
effective  decisions  [emphasis  added].  The  primary  goal  of  test 
programs  should  be  to  experiment,  learn  about  the  strengths 
and  weaknesses  of  newly  added  capabilities  or  (sub)systems, 
and  use  the  results  to  improve  overall  system  performance. 
Furthermore,  data  from  previous  stages  of  development, 
including  field  data,  should  be  used  in  design,  development, 
and  testing  at  future  stages,  (p.  3) 

The  ATEC  aerial  ISR  SoS  evaluation  example  described  earlier 
followed  these  lessons  learned.  The  aerial  ISR  SoS  was  an  evolution¬ 
ary  system,  and  all  testing  and  evaluation  activities  were  integrated 
into  the  overall  evaluation.  Additionally,  M&A  was  used  to  extend  the 
evaluation  to  examine  and  explore  the  other  possible  employment 
combinations  and  methods  for  this  SoS  in  an  asymmetric  insurgent 
environment. 


Conclusions 

The  importance  and  implementation  of  thorough  SoS  evalua¬ 
tion-distributing  and  measuring  the  effects  of  individual  systems  as 
they  are  integrated  across  the  entirety  of  the  SoS— poses  a  challenge 
to  the  test  community.  This  challenge  can  be  met  by  incorporating 
the  lessons  learned  and  data  from  multiple  events,  using  M&A  and 
scientific  methodology  to  integrate  and  optimize  the  testing  and  pro¬ 
viding  relevant  feedback  to  affected  communities— typically  soldiers 
employing  the  SoS  or  planners  executing  acquisition  strategy.  With 
the  continued  interdependence  of  SoS  deployed  and  relied  upon  in 
operational  environments,  SoS  evaluation  must  be  capable  of  using  all 
available  assets  to  ensure  operational  realism  is  met  in  all  test  events, 
and  relevant  quantitative  measures  are  applied  in  evaluating  the  SoS, 
ensuring  a  legitimate  SoS  evaluation,  not  merely  an  evaluation  of  a 
system  within  a  system. 
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PERFORMANCE-BASED 
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EFFECTIVE  GOVERNMENT 
PARTICIPATION 

(  Steve  Geary,  Scott  Koster,  Wesley  S.  Randall, 
and  Jeffrey  J.  Haynie 

Organic  government-owned  and  -managed  product  support 
organizations  are  often  viewed  as  less  capable  than  their 
commercial  counterparts.  In  fact,  highly  effective  government 
organization  participants  in  product  support  do  exist,  supported 
by  a  host  of  success  enablers  in  use  at  government-owned  and 
-managed  organizations  across  the  Services.  These  enablers 
can  stimulate  best-value  participation  by  government  orga¬ 
nizations  in  performance-based  life  cycle  support  strategies. 

More  effective  government  participation  results  in  increased 
synergy  and  collaboration  for  the  warfighter,  the  organic  struc¬ 
ture,  and  the  taxpayer.  This  article  documents  and  describes 
some  of  the  success  enablers  used  to  catalyze  more  effective 
integration  of  the  government-managed  support  structure 
into  the  industrial  base. 


Keywords:  Product  Support,  Product  Support 
Strategy,  Life  Cycle  Product  Support  Strategies, 
Performance-Based  Logistics  (PBL),  Product 
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In  2009  the  Department  of  Defense  (DoD)  published  a  report  on 
Product  Support  Assessment  (DoD,  2009),  with  particular  emphasis 
on  DoD’s  vision  for  improving  the  integration  of  government-owned 
and  -managed  capabilities  into  performance-based  product  support 
strategies.  Rather  than  treating  the  government  share  of  the  industrial 
base  as  distinct  from  the  commercial  base,  the  report  develops  and 
posits  the  notion  of  a  single  industrial  base,  partially  managed  by  the 
government  and  partially  managed  by  the  commercial  sector;  and 
that  the  government  participants  can  be  best-value  participants.  The 
vision  for  industrial  integration  strategy  uses  this  foundation  to  speak 
to  the  opportunity  for  synergy  from  a  more  collaborative  organic  and 
commercial  industrial  base. 

Effective  product  support  requires  contributions  from  both  the 
public  and  private  sectors.  A  significant  challenge  over  the  course 
of  the  next  decade,  particularly  in  today’s  acquisition  environment 
of  declining  financial  resources  combined  with  projected  deficits 
and  undiminished  operational  demands,  is  creating  a  more  effec¬ 
tive,  unified,  and  fiscally  prudent  industrial  integration  strategy  for 
product  support.  More  than  60  years  after  World  War  II  (WWII), 
when  the  standing  commercial  industry  (still  seen  today)  originally 
spurred  the  post-World  War  II  economic  boom,  the  DoD  has  yet  to 
fully  leverage  and  blend  the  knowledge,  skills,  and  capabilities  of  the 
complete  defense  industrial  base  through  a  considered  and  deliber¬ 
ate  integration  strategy. 

As  a  part  of  the  continuing  efforts  to  achieve  acquisition  reform, 
Congress  has  passed  legislation  better  defining  the  role  of  govern¬ 
ment  entities  involved  in  executing  product  support  strategies.  The 
government  has  always  been  fully  responsible  and  accountable  for 
product  support  delivered  to  the  warfighter.  That  principle  has  been 
reinforced  with  the  passage  of  the  National  Defense  Authorization  Act 
of  FY  2010  (NDAA,  2009).  Section  805  of  the  NDAA  adds  clarity  to 
and  elaborates  on  this  principle. 

The  provisions  of  Section  805  require  that  the  Secretary  of 
Defense  issue  guidance  on  life  cycle  management  and  the  imple¬ 
mentation  of  product  support  strategies  for  major  weapon  systems. 
Additionally,  each  major  weapon  system  shall  have  a  product  support 
manager  to  develop,  implement,  and  validate  the  product  support 
strategy  (e.g.,  Performance-Based  Logistics  [PBL],  sustainment  sup¬ 
port,  contractor  logistics  support ,  life  cycle  product  support,  or 
weapon  system  product  support). 

The  responsibility  for  the  product  support  strategy  is  clearly  in 
the  hands  of  the  government.  In  addition,  most  government  partici¬ 
pation  in  product  support  is  more  expansive  than  oversight.  Some 
product  support  must  be  performed  by  the  government.  To  cite  two 
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common  examples,  either  statutory  requirements  or  operational 
requirements  in  forward-deployed  environments  dictate  execution 
of  certain  tasks  by  the  military.  Other  examples  demonstrate  that 
government  organizations  are  best-practice  contributors  to  product 
support  because  their  role  and  participation  are  driven  by  best  value, 
not  statutory  requirement. 

Across  the  landscape  of  performance  outcome-based  product 
support  strategies,  numerous  examples  illustrate  the  adoption  of  best 
practices  that  allows  government-owned  and  -managed  capabilities 
to  participate  in  product  support  strategies  as  best-value  contribu¬ 
tors.  These  examples  demonstrate  an  ability  to  overcome  commonly 
cited  obstacles  to  participation  by  government  elements  and  execute 
a  more  successful  integration  of  the  organic  assets  into  a  unified 
industrial  base.  Although  it  is  true  that  government  organizations 
are  not  profit-making  businesses,  they  are  businesses,  nonetheless, 
and  can  successfully  compete  and  win  in  PBL  using  best-in-class 
practices. 

In  this  article,  we  will  examine  four  distinct  case  studies  that 
demonstrate  a  spectrum  of  viable  practices  available  to  government 
organizations  to  allow  them  to  compete  on  merit  for  business  as  prod¬ 
uct  support  integrators  and  product  support  providers.  A  description 
of  the  four  case  studies  follows: 

Case  Study  No.  1:  An  Environment  of  Success,  Huntsville 

Case  Study  No.  2:  From  Source  of  Repair  to  Business 
Partner,  Jacksonville 

Case  Study  No.  3:  The  Joint  STARS  Contract-A  Decade  of 
Success 

Case  Study  No.  4:  The  Upstarts-Naval  Surface  Warfare 
Center,  Crane  Division 

The  foundational  development  of  core  competencies  through  the 
incubation  of  best-practice  capabilities,  as  envisioned  in  10  U.S.C., 
Section  2474,  makes  this  possible  (Armed  Forces,  2004): 

The  Secretary  of  Defense  shall  establish  a  policy  to  encourage 
the  Secretary  of  each  military  department  and  the  head  of 
each  Defense  Agency  to  reengineer  industrial  processes  and 
adopt  best-business  practices  at  their  Centers  of  Industrial 
and  Technical  Excellence  in  connection  with  their  core  com¬ 
petency  requirements,  so  as  to  serve  as  recognized  leaders 
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in  their  core  competencies  throughout  the  Department  of 
Defense  and  in  the  national  technology  and  industrial  base. 

From  a  financial  standpoint,  effective,  efficient,  and  best-value 
use  of  government-owned  resources  is  a  victory.  The  U.S.  taxpayers 
have  a  huge,  long-standing  investment  in  government-owned  sup¬ 
port  capabilities,  particularly  in  inventory  control,  distribution,  and 
maintenance  depots.  At  the  same  time,  although  the  government- 
owned  and  -managed  base  contributes  significantly,  it  cannot  do  all 
of  DoD’s  product  support  work.  American  industry  provides  a  source 
of  innovation,  and  flexible  and  productive  capacity  for  the  defense 
industrial  base.  The  way  ahead  for  more  cost-effective  product  sup¬ 
port  lies  in  effective  blending  of  these  complementary  capability  sets 
where  the  best  use  is  made  of  the  entire  industrial  base,  facilitated 
by  the  continuing  expansion  of  best  business  practices  in  both  the 
commercial  and  government  sectors. 

We  should  not  lose  sight  of  the  fact  that  more  effective  govern¬ 
ment  participation  in  DoD’s  product  support  strategies  will  result  in 
better  collaboration  and  synergy  among  the  warfighters,  the  organic 
structure,  and  U.S.  taxpayers. 


Method 

The  study  team  followed  a  simple  three-step  process  to  produce 
the  resultant  case  studies.  The  first  step  consisted  of  identifying 
candidate  programs  with  characteristics  that  were  germane  to  the 
research  objectives.  The  second  step  was  researching  and  inter¬ 
viewing  representatives  from  the  candidate  programs.  The  final 
step  included  analyzing  and  writing  the  study  findings  that  form 
the  basis  of  this  article. 

To  identify  and  select  potential  candidate  programs  for  this 
study,  the  team  used  two  primary  criteria  and  one  limiting  fac¬ 
tor.  The  primary  criteria  are  the  location  in  the  Decision  Matrix  for 
Product  Support,  or  DMPS  (Figure  1)  and  distinctive  performance. 
In  selecting  the  programs  to  review,  the  limiting  factor  was  the  avail¬ 
ability  of  program  sustainment  teams  to  support  the  inquiry.  The 
team  looked  across  the  Services  and  at  end-item  operating  environ¬ 
ments  (land,  sea,  air,  and  space)  as  a  consideration  in  the  selection 
of  candidate  programs  to  include. 

Understanding  how  the  candidate  programs  populate  the  DMPS 
in  Figure  1  enabled  the  team  to  base  the  selection  decision  on  the 
characteristics  of  the  product  support  strategies  from  an  objective 
perspective.  In  short,  the  DMPS  was  designed  to  help  program  man- 
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FIGURE  1.  DECISION  MATRIX  FOR  PRODUCT  SUPPORT 


r  System 


4-  Subsystem 


Component 


Note.  Adapted  from  DoD  Weapon  Systems  Acquisition  Reform  Product  Support 
Assessment,  Department  of  Defense,  2009,  p.  46.  Retrieved  from  https://acc.dau.mil/ 
CommunityBrowser.aspx?id=328610.  HIMARS  =  High  Mobility  Artillery  Rocket  System; 
APU  =  Auxiliary  Power  Unit;  USAF  IPV  =  U.S.  Air  Force  Industrial  Prime  Vendor. 

agers  identify  their  product  support  strategy.  A  program’s  location 
in  the  matrix  will  influence  decisions  relative  to  the  Product  Sup¬ 
port  Integrator  (PSI)  composition,  metrics,  incentives,  Performance 
Based  Agreement  (PBA),  and  analytical  tools.  The  matrix  is  based  on 
a  framework  that  outlines  nine  product  support  options  as  defined 
by  the  intersection  of  two  key  strategic  system  characteristics  that 
drive  the  appropriate  support  strategy.  The  key  strategic  system 
characteristics  are  weapon  system  strategy  and  integration  strat¬ 
egy  (DoD,  2009).  The  two  characteristics  are  useful  mechanisms 
to  categorize  programs  so  that  the  team  can  focus  on  only  those 
programs  that  are  in  line  with  the  study  objectives. 

Figure  1  is  also  used  to  identify  those  programs  with  “blended” 
integration  strategies.  From  the  population  of  programs  with  a 
“blended”  integration  strategy,  the  team  looked  for  programs  from 
each  of  the  weapon  system  strategies.  With  the  limiting  factor  of 
program  availability,  the  study  team  was  able  to  identify  candidate 
programs  in  two  of  the  three  weapon  system  strategy  categories 
of  subsystem  and  system. 

From  this  list  of  candidate  programs,  the  study  team  next  looked 
for  discriminating  factors  to  identify  five  or  six  programs  that  formed 
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Figure  2.  A  U.S.  sailor  assigned  to  the  aviation  intermediate  maintenance  department's 
jet  shop  watches  during  a  jet  engine  test  cell  on  an  F404-GE-400  jet  engine  for  an  F/A- 
18C  Hornet  aircraft  on  the  fantail  of  aircraft  carrier  USS  Nimitz  (CVN  68),  October  14, 
2009,  in  the  Indian  Ocean.  The  F404  engine  has  twice  won  the  Department  of  Defense 
Performance-Based  Logistics  Award.  U.S.  Navy  photo  by  MC3  John  Phillip  Wagner  Jr. 


the  target  programs  to  review.  Discriminating  factors  included  rec¬ 
ognition  of  excellence  (DoD  PBL  submission  packages),  duration  of 
current  “blended”  product  support  strategy,  “commerciality”  of  the 
materiel,  and  ongoing  research  efforts  at  Redstone  Arsenal,  Alabama. 

The  selection  process,  in  conjunction  with  program  availabil¬ 
ity,  resulted  in  the  selection  of  three  programs  for  the  research 
and  interview  step.  The  study  team  gathered  information  on  the 
programs  and  traveled  to  the  program  offices,  depot  business 
offices,  and  remanufacturing  facilities  to  interview  the  managers  and 
artisans  involved  in  each  project.  The  team  also  incorporated  organi¬ 
zational  team  climate,  based  upon  related  research  between  Auburn 
University  and  U.S.  Army  Air  and  Missile  Command  (AMCOM)  in 
Huntsville,  Alabama. 

With  the  time  available,  the  team  was  able  to  perform  a  “deep 
dive”  on  the  F404  engine,  which  has  twice  won  the  DoD  PBL  award 
(Figure  2).  Product  support  awards  are  a  result  of  the  remanufactur¬ 
ing  activity  at  the  Fleet  Readiness  Center,  Southeast  (FRCSE),  and 
subsystem  inventory  management  at  the  Defense  Logistics  Agency. 
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Case  Study  No.  1: 

An  Environment  for  Success,  Huntsville 

The  U.S.  Army  AMCOM  has  created  an  environment  of  high- 
performing,  award-winning  product  support  teams.  Huntsville 
organizations  have  earned  recognition  for  their  performance-based 
product  support  initiatives.  Specifically,  a  number  of  Huntsville 
programs  have  won  the  annual  Secretary  of  Defense  Award  for 
Excellence  in  PBL,  as  follows: 

2005  Shadow  200  Tactical  Unmanned  Aircraft  System 

2006  High  Mobility  Artillery  Rocket  System 

2007  Improved  Target  Acquisition  System 

2008  Tactical  Airspace  Integration  System 

2009  High  Mobility  Artillery  Rocket  System 

In  fact,  this  is  the  complete  list  of  PBL  Award  winners  for  the 
Army,  and  every  one  of  these  award  winners  is  at  Huntsville.  No 
Army  competitor  for  the  PBL  Award  has  ever  won  the  competition 
from  any  location  other  than  Huntsville.  According  to  the  2009  PBL 
Award  memo: 

Performance-Based  Logistics  (PBL)  is  the  Department  of 
Defense's  strategy  to  improve  weapon  system  readiness  by 
obtaining  life  cycle  product  support  of  weapon  systems,  sub¬ 
systems,  and  components  as  an  integrated  package  based 
on  output  measures,  such  as  materiel  availability,  materiel 
reliability,  and  reduced  ownership  cost. 

The  Secretary  of  Defense  PBL  Awards  recognize  gov¬ 
ernment  and  industry  teams  that  have  demonstrated 
outstanding  achievements  in  providing  our  warfighters  with 
exceptional  operational  capability  through  PBL  agreements. 
(Carter,  2009) 

What  makes  Huntsville  distinctive?  How  can  it  so  completely 
dominate  as  the  Army’s  leader  in  PBL?  What  special  enablers  are 
present  in  the  Huntsville  environment?  Why  is  Huntsville  so  successful 
in  driving  outcome-based  product  support  strategies  that  maxi- 
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mize  contributions  from  across  the  industrial  base  (Haynie,  Randall, 
Armenakis,  &  Geary,  2009). 

In  this  phase  of  the  research,  we  sought  to  understand  and  identify 
the  practices  that  contribute  to  this  high  performance.  Accordingly, 
we  interviewed  team  members  from  many  of  the  high-performing 
product  support  teams  from  AMCOM,  including  many  of  the  award 
winners.  These  interviews  included  personnel  from  both  govern¬ 
ment  and  industry  who  were  involved  with  post-production  support 
of  U.S.  and  allied  defense  systems.  We  ensured  that  our  interviews 
included  strong  government  participation  by  conducting  interviews 
with  personnel  from  Apache,  Letterkenny  Army  Depot,  Close  Combat 
Weapon  Systems,  Corpus  Christi  Army  Depot,  Unmanned  Aerial  Sys¬ 
tems  Logistics  Division,  the  Integrated  Materiel  Management  Center, 
and  the  Precision  Fires  Project  Office. 

Through  discussion  during  site  visits,  we  attempted  to  understand 
what  behaviors  and  perceptions  led  to  success.  Those  we  interviewed 
gave  us  a  rich  perspective  of  the  inter  and  intraorganizational  behav¬ 
iors  that  appear  to  foster  PBL  success. 

We  found  individual  behaviors  and  organizational  processes 
were  consistent  with  suggestions  in  supply  chain  research  on  how  to 
improve  collaborative  performance.  What  we  found  was  that  AMCOM 
appears  to  have  fostered  a  PBL  culture  that  aligned  and  oriented 
those  behaviors  and  processes.  The  orientation  created  through  a 
proactive  PBL  strategy  appears  to  positively  influence  cost  control 
and  performance  improvement  in  the  eyes  of  the  managers.  The 
strength  of  performance-based  support  strategies  seems  to  be  their 
ability  to  strategically  align  cross-functional  and  interorganizational 
processes  of  multiple  firms,  customers,  and  bill  payers;  and  focus 
them  on  a  long-term  performance  goal  in  a  manner  that  creates 
consistent  and  measurable  success.  The  review  of  the  environment 
at  Huntsville  was  not  intended  to  validate  the  efficacy  of  their  per¬ 
formance-based  approach.  Rather,  given  the  numerous  PBL  Award 
winners  from  Huntsville,  the  research  uncovered  eight  factors  critical 
to  enabling  an  environment  for  success. 

Eight  Critical  PBL-Driven  Interorganizational  Success  Factors 

In  effect,  adoption  of  a  performance-based  product  support 
approach  represents  a  strategic  change  in  interfirm  practice.  By 
analyzing  teams  that  implemented  this  new  strategy  of  outcome- 
based  product  support,  we  found  key  enabling  factors  present  in 
the  environment  that  contributed  to  the  successful  participation  in, 
and  often  leadership  of,  outcome-based  programs  by  government 
organizations. 
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The  eight  factors  identified  at  Huntsville  include: 


FACTOR  NO.  1 

Cooperative  Interdependence.  Cooperative  interdependence  is  an 
understanding  that  goal  attainment  is  dependent  upon  other  team 
members  reaching  their  goals  (Deutsch,  1973). 


FACTOR  NO.  2 

Transformational  Leadership.  Transformational  leaders  transcend 
short-term  goals  (Eisenbeiss,  Van  Knippenberg,  &  Boerner,  2008) 
and  focus  their  attention  on  the  higher  order  intrinsic  needs  of  sub¬ 
ordinates,  inducing  them  to  transcend  their  own  self-interests  for  the 
benefit  of  the  organization  or  team. 


FACTOR  NO.  3 

Team  Climate  for  Innovation.  Team  vision,  participative  safety,  cli¬ 
mate  for  excellence,  and  support  for  innovation  are  components  in  the 
creative  process,  leading  to  greater  team  innovation  (Deutsch,  1973). 


FACTOR  NO.  4 

Team  Innovation.  Team  innovation  is  the  combination  of  the  qual¬ 
ity  and  quantity  of  creative  ideas  that  have  been  implemented  within 
an  organization.  These  innovations  represent  changes  and  can  be 
either  administrative  or  technological  in  nature  (West,  2002). 


FACTOR  NO.  S 

Team  Learning.  Team  learning  is  the  process  by  which  teams 
discuss  and  solve  problems.  Collectively,  the  team  engages  in  informa¬ 
tion  seeking  and  reflective  decision-making  processes  that  positively 
impact  the  degree  of  knowledge  and  information  for  other  members 
(Hirst,  Van  Knippenberg,  &  Zhou,  2009). 


FACTOR  NO.  6 

Team  Performance.  Objective  performance  represents  the  out¬ 
comes  of  the  team’s  activities  that  are  valued  by  one  or  more  of 
its  constituencies,  such  as  reductions  in  operating  costs,  greater 
efficiency,  and  increases  in  profits  (Mathieu,  Maynard,  Rapp,  & 
Gilson,  2008). 
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FACTOR  NO.  7 

Change  Appropriateness.  Innovations  can  produce  desired  out¬ 
comes  such  as  increased  product  quality  and  reduced  support  costs. 
However,  it  is  important  that  the  appropriateness  of  the  innovation  be 
taken  into  account;  unbridled  innovation  can  be  counter-productive 
(Armenakis,  Harris,  Cole,  Fillmer,  &  Self,  2007). 


FACTOR  NO.  8 

Means  Efficacy  Climate.  Means  efficacy  climate  is  the  shared 
attitude  concerning  the  degree  of  organizational  support  supplied  to 
the  team  through  policies,  processes,  and  procedures  (Eden,  2001). 

These  enablers  are  not  new.  They  are  described  and  validated  in 
the  more  generalized  academic  literature  related  to  management  and 
change  management.  What  is  new  is  the  manner  in  which  these  fac¬ 
tors  interrelate  under  a  PBL  strategy  to  create  an  environment  whose 
whole  is  greater  than  the  sum  of  its  parts.  Our  findings  demonstrate 
how  a  proactive  PBL  strategy  provides  a  benchmark  of  management 
best  practices. 

To  close  the  loop,  the  research  team  conducted  validation  sessions 
with  senior  executives,  senior  managers,  engineers,  program  manag¬ 
ers,  and  logisticians  familiar  with  performance-based  strategies.  The 
subject  matter  experts  confirmed  that,  from  their  perspective,  the 
data,  analysis,  and  identified  factors  fit  with  their  environments. 

Implications  for  Outcome-Focused  Product  Support  Success 

The  AMCOM  has  a  culture  of  demonstrating  innovation  and  lead¬ 
ership  when  it  comes  to  post-production  support.  Our  research 
suggests  that  an  organization-wide  understanding  of  the  eight  fac¬ 
tors  that  interrelate  to  create  a  winning  culture  drives  success.  These 
factors  have  evolved  and  emerged  at  Huntsville  largely  because  they 
have  been  able  to  create  a  collaborative  partnership  approach.  This 
partnership  extends  across  the  organization  and  into  partners  in 
the  industrial  base  and  is  less  adversarial  in  style,  based  on  a  mutual 
understanding  of  where  the  motivations  and  interests  of  each  party 
lie.  By  acknowledging  and  managing  areas  of  divergence  and  ten¬ 
sion,  and  supported  by  a  willingness  to  share  information  in  a  spirit  of 
openness  and  transparency  at  all  levels,  the  partnership  philosophy 
becomes  a  key  competency.  Creation  of  that  environment  must  take 
place  within  the  government  post-production  support  infrastructure, 
culture,  and  resources  in  order  to  drive  PBL.  The  AMCOM  in  Huntsville 
identifies  and  demonstrates  the  elements  that  are  best  practices. 
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Expanding  the  role  of  DoD’s  government-run  sustainment  infra¬ 
structure  (e.g.,  depots)  presents  challenges  in  the  planning  for,  and 
delivery  of,  integrated,  affordable,  outcome-focused  product  support. 
What  Huntsville  demonstrates  is  that,  regardless  of  the  obstacles, 
critical  success  factors  are  known  and  are  within  the  control  of  those 
leaders  responsible  for  the  government’s  post-production  support 
infrastructure.  Good  management  drives  performance-based  success; 
this,  in  turn,  leads  to  a  win  for  the  warfighter,  the  organic  structure, 
and  the  taxpayer. 

Case  Study  No.  2:  From  Source  of  Repair  to 
Business  Partner,  Jacksonville 

Jacksonville's  Journey 

“We  provide  aviation  maintenance  solutions  that  satisfy  Navy 
warfighter’s  demands,”  according  to  the  FRCSE  mission  statement. 
Actually,  from  a  review  of  two  product  support  efforts,  the  FRCSE, 
located  in  Jacksonville,  Florida,  is  evolving  by  leveraging  its  robust 
manufacturing  capability  and  forward-leaning  business  practices 
that  help  position  existing  capacity  for  use  in  partnerships.  The  new 
development  is  the  extent  that  FRCSE  and  its  private-sector  business 
partners  have  aligned  their  respective  business  models  to  create  a 
blended  and  compelling  value  proposition  for  the  warfighter. 

Like  Huntsville,  Jacksonville  has  created  an  innovative  environ¬ 
ment  where  the  adoption  and  application  of  best  commercial  business 
practices  have  been  embraced.  The  discovery  work  at  Huntsville 
included  a  detailed  validation  of  the  elements  required  to  develop 
a  working  environment,  receptive  to  the  adoption  of  best  practices 
in  support  of  performance-based  life  cycle  product  support.  Rather 
than  replicating  Huntsville,  the  review  in  Jacksonville  focused  on  the 
implementation  of  specific  best  practices  themselves. 

High-visibility,  performance-based  product  support  strategies  are 
currently  in  use  by  several  programs  in  Jacksonville.  The  two  prod¬ 
uct  families  that  participated  in  the  research  are  the  F404  and  F414 
engines,  used  on  the  F-18  aircraft;  and  the  Forward  Looking  Infrared 
family  of  sensors,  used  on  a  variety  of  platforms.  Meetings  took  place 
with  the  FRCSE  business  office,  as  well  as  company  representatives 
from  General  Electric  Aircraft  Engines  (GEAE)  and  Raytheon,  the 
respective  Original  Equipment  Manufacturer  (OEM)  and  business 
partner  on  these  programs. 
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Foundation 

For  both  of  these  programs,  the  following  four  solid  foundational 
elements  of  performance-based  product  support  set  down  in  the 
public-private  partnership  fully  align  with  the  description  offered  in 
the  Product  Support  Assessment  (DoD,  2009). 


ELEMENT  NO.  1 

Long-term  committed  relationships  executed  with  flexibility  and 
integrated  across  organizational  boundaries,  with  complementary 
skill  sets  and  abilities,  are  both  essential  and  possible. 


ELEMENT  NO.  2 

Shared  partnership  vision  and  objectives  with  the  right  metrics 
and  incentives  drive  alignment  and  are  especially  effective  when 
supported  by  a  clear  delineation  of  complementary  roles  and  respon¬ 
sibilities. 


ELEMENT  NO.  3 

Full  coordination  with  all  stakeholders  supported  by  transparency, 
open  communication,  and  the  flexibility  to  change  partnership  scope 
is  an  essential  ingredient  to  success. 


ELEMENT  NO.  4 

Clearly  documented  objectives  support  alignment  and  fuel  the 
success  of  the  partnership.  This  can  be  achieved  through  incentives 
that  drive  desired  outcomes  and  are  supported  by  sound  economic 
analysis. 

Nothing  new  or  particularly  innovative  is  embodied  in  these 
foundational  elements  of  performance-based  product  support.  The 
Government  Accountability  Office  (GAO,  2003)  first  documented  the 
essential  elements.  What  is  interesting  at  Jacksonville  is  the  distinc¬ 
tive  degree  of  integration  and  coordination  they  have  established  on 
top  of  this  foundation. 

Beyond  Transactional  Approaches  to  Maintenance,  Repair,  and  Overhaul: 

Moving  From  Wrench  Turning  to  an  Integrated  Business  Model 

One  of  the  recommendations  of  the  Product  Support  Assessment 
(DoD,  2009)  speaks  to  the  vision  of  leveraging  government-managed, 
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post-production  support  capabilities  outside  of  the  traditional,  pro¬ 
gram-centric  events: 

...Expand  partnering  ‘beyond  maintenance,’  drive  standardiza¬ 
tion  across  Services,  and  promote  proactive  establishment  of 
single-source  repair  capability,  (p.  43) 

At  Jacksonville,  we  see  in  execution  a  prototype  defense  industrial 
base  of  the  future.  Here,  a  government-run  depot  support  operation 
has  moved  beyond  a  job-shop  maintenance  operation,  becoming  a 
fully  capable  industrial  partner  that  is  deeply  integrated  with  com¬ 
mercial  partners.  The  partnerships  that  are  being  created  provide 
the  government  customer  with  a  unified  government  and  industry 
post-production  support  effort.  The  individual  activities  in  large  part 
are  not  distinctive,  but  the  degree  of  integration  and  coordination  is. 

In  the  engine  shop,  under  one  roof,  the  Jacksonville  support 
operation  maintains  engines  from  two  different  OEMs.  Integrated  pro¬ 
cesses  and  shared  capacities  support  both  OEM  families,  managed 
by  a  unified  staff.  Through  this  Navy  capability  set,  it  also  maintains 
engines  for  the  A-10  aircraft— an  Air  Force  platform.  The  Jacksonville 
operation  is  moving  inexorably  down  the  path  of  managing  its  engine 
maintenance  capability  in  a  standardized  fashion  across  product 
families,  and  indeed  across  Services. 

Its  success  has  led  to  the  capture  of  additional  work  from  GEAE, 
formerly  performed  at  the  GEAE  facilities  north  of  Boston.  This  is 
the  typical  pattern  of  success  in  the  maintenance  arena  for  depot 
partnerships.  Yet,  in  Jacksonville,  this  accretion  of  additional  work 
has  created  an  opportunity  to  move  beyond  legacy  maintenance 
functions.  The  facility  is  now  being  audited  by  GEAE  to  become,  in 
addition  to  the  current  role  as  a  source  of  repair  for  the  404  and  414 
engines,  a  new  module  manufacturing  site  for  the  414  engine. 

The  FLIR  team,  including  both  Raytheon  and  the  FRCSE,  also 
demonstrates  highly  evolved  thought  beyond  the  traditional  mainte¬ 
nance  partnership  roles.  In  discussions,  they  clearly  draw  a  distinction 
between  maintenance  partnerships  and  business  partnerships;  for 
the  FLIR  sensors,  the  team  maintains  that  they  are  in  a  business 
partnership  and  that  they  have  moved  beyond  wrench  turning  a 
long  time  ago. 

The  original  PBL  in  the  FLIR  family  was  for  the  device  on  the  H-60 
helicopter.  Rather  than  viewing  this  as  a  unique  opportunity,  first 
Raytheon,  and  then  Raytheon  in  partnership  with  the  FRCSE,  saw 
this  as  a  competitive  opportunity  to  capture  more  work.  Over  time, 
capacities  and  equipment  were  upgraded  in  Jacksonville.  With  Ray¬ 
theon  as  the  prime  and  the  FRCSE  as  a  teammate  and  subcontractor, 
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the  team  competed  for  and  earned  additional  work.  Today,  a  single 
set  of  equipment  in  one  government  building  services  a  diverse  set 
of  FLIR  devices. 

A  breakthrough  took  place  in  2009.  The  capacities  in  Jack¬ 
sonville  were  purposely  designed  to  be  able  to  accommodate  the 
FLIRs  installed  on  the  Air  Force’s  Predator  and  Reaper  Unmanned 
Aerial  Vehicle  platforms.  The  ability  to  maintain  these  FLIRs  is  a  core 
requirement,  meaning  that  the  DoD  is  statutorily  required  to  maintain 
government  post-production  support  capability.  The  Navy’s  FRCSE 
was  selected  by  the  Air  Force  as  the  Depot  Source  of  Repair  for  the 
Predator  and  Reaper  FLIRs.  The  complete  set  of  FLIRs  that  are  now 
slotted  to  use  this  capacity  are  AAS-44V  (older  H-60  series),  AAS- 
44(C)V  (MTS-A  for  H-60R/S),  AAS-52  (MTS-A  for  Predator),  DAS-1 
(MTS-B  for  Reaper),  AAQ-27  (MV-22),  and  the  AAQ-29  (CH-53E). 

By  viewing  themselves  as  an  integrated  capability  set,  the  FRCSE 
and  various  industry  teams  have  been  able  to  step  beyond  traditional 
program-centric  maintenance  relationships.  They  are  now  integrat¬ 
ing  horizontally  across  the  portfolio,  and  they  are  integrating  across 
the  Services.  The  government’s  post-production  support  capabili¬ 
ties,  developed  under  the  umbrella  of  Section  2474  and  nurtured 
by  their  industry  partners,  are  stepping  into  higher  level  activities 
like  new  module  assembly.  Private  industry  has  been  instrumental  in 
directly  assisting  the  incorporation  of  these  best  practices  into  this 
public  facility,  and  together  the  team  is  reaping  the  benefit.  They  are 
bootstrapping  themselves  through  an  evolutionary  process  toward 
becoming  a  single-source  capability  for  specific  technologies  used 
across  the  Services. 


Enabling  Best  Practices 

The  FRCSE  has  demonstrated  an  ability  to  deploy  a  broader 
approach  to  partnership  that  is  not  the  typical  public-private  partner¬ 
ship  based  on  arms-length  arrangements.  This,  in  turn,  has  allowed 
industry  to  look  to  the  government  post-production  support  infra¬ 
structure  for  a  capability  that  is  sought  by  the  industrial  base.  With 
the  jet  engine,  that  means  the  FRCSE  can  provide  jet  engine  fabrica¬ 
tion  and  assembly  expertise,  not  just  artisan  labor.  For  the  FLIR,  it 
means  the  FRCSE  has  off-the-shelf,  one-stop  capacity  and  capability 
to  perform  maintenance,  repair,  and  overhaul  on  a  technology  that 
is  becoming  more  and  more  ubiquitous  and  sophisticated  across 
military  weapon  systems. 

This  evolution  did  not  happen  overnight.  According  to  the 
research  participants  familiar  with  the  progression,  the  team  built  a 
foundation  of  business  partnerships  based  upon  a  common  strategic 
vision.  Each  party  identified  their  revenue  streams  and  began  work- 
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ing  toward  a  “business”  relationship  that  addressed  the  needs  of  each 
participant.  This  developmental  process  spanned  years. 

These  needs  are  different  from  the  point  of  view  of  Jacksonville 
and  the  industrial  partners.  For  the  industrial  partners,  the  definition  of 
need  is  simple:  profit.  Real  dollars  flowing  to  the  bottom  line  matter  to 
commercial  organizations.  On  the  other  hand,  the  government  depots, 
and  other  post-production  support  organizations,  are  “break  even” 
operations.  The  FRCSE  looks  at  sustaining  or  increasing  labor  hours, 
avoiding  Base  Realignment  and  Closure  recommendations,  satisfying 
statutory  requirements  (core,  50/50,  etc.),  and  improving  support  of 
the  fleet  as  “profit.” 

Rather  than  clashing  over  the  differing  needs,  the  FRCSE  has  found 
common  ground  that  allows  it  to  operate  in  understandable  “swim  lanes” 
with  its  industrial  partners.  The  FRCSE  lauds  industry’s  superior  ability  to 
manage  component  supply  more  effectively,  and  unhesitatingly  turns  to 
industry  to  contribute.  Other  areas  where  the  FRCSE  accepts  help  are: 
technical  data,  information  systems,  test,  training,  technical  assistance, 
transportation,  packaging,  engineering  analysis,  inventory  management, 
quality  support,  logistical  services,  materiel  movements,  and  engineering 
on  the  shop  floors.  Although  the  FRCSE  and  its  industrial  partners  are 
grappling  with  a  complicated  set  of  best-value  decisions,  none  of  them 
retreat  from  making  the  hard  business  decisions. 

Motivated  by  self-interest,  FRCSE  and  its  partners  have  maneu¬ 
vered  themselves  into  a  position  where  they  focus  together  on  the  joint 
opportunities  and  seek  to  grow  the  business  and  consequential  benefits 
to  each  party.  This  is  an  extremely  sophisticated,  strategic  approach 
to  business.  Or,  as  one  industrial  partner  described  the  process,  they 
worked  diligently  to  “put  the  depot  in  a  position  that  they  would  have 
to  make  a  bad  business  decision  by  not  forming  a  true  business  part¬ 
nership.” 

In  conjunction  with  the  development  of  a  shared  strategic  vision,  the 
government  post-production  support  activities  implicitly  adopt  a  mind¬ 
set  that  drives  alignment  to  the  desired  outcomes.  The  introduction  of 
performance  into  the  equation  encourages  the  OEMs  to  competitively 
seek  to  meet  the  benchmarks,  and  to  find  partners  who  can  help  them 
do  it.  This,  in  turn,  encourages  the  government  post-production  support 
organizations  to  improve  in  areas  where  they  have  competency,  thereby 
making  them  more  attractive  to  the  OEMs.  This  creates  a  positive, 
perpetual  cycle  that  drives  best  practices  into  the  government  post¬ 
production  support  organizations,  all  resulting  from  the  embrace  of  a 
shared  strategic  vision. 

Highly  visible  indicators  are  evidence  of  the  depth  of  alignment  and 
integration  between  commercial  partners  and  the  government  post-pro¬ 
duction  support  industrial  base.  Technical  employees  of  the  commercial 
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partners  are  embedded  within  the  government  post-production 
support  operation,  including  on-site  offices  and  free  access  to  the 
workspaces  of  the  artisans.  The  FRCSE  has  embraced  Lean  and  Six 
Sigma  approaches  to  continuous  improvement.  Bulletin  boards  are 
prominently  displayed  with  objective  performance  measures  so  all 
employees  can  see  what  they  are  being  measured  against. 

Further,  contrary  to  conventional  wisdom,  artisans  can  earn 
incentive  payments  based  on  their  performance.  In  the  contemporary 
financial  environment,  cost  reduction  is  an  imperative  in  any  PBA.  So, 
although  the  FRCSE  works  on  a  cost-reimbursable  basis,  it  has  put 
in  place  a  very  aggressive  gainsharing  program  with  the  artisans— in 
a  union  environment  no  less.  A  “controllable”  hourly  labor  cost  is 
defined  for  each  work  center,  and  40  percent  of  any  achieved  cost 
reduction  against  that  rate  is  paid  to  the  employees.  For  reimburse¬ 
ment  purposes,  the  depot  can  still  invoice  for  the  incentives  paid, 
because  the  bonuses  are  considered  labor  cost,  but  the  achieved 
hourly  cost  reduction  rolls  into  the  controllable  hourly  rate  for  the 
next  reporting  period. 


Open  Issues 

The  FRCSE  follows  conventional  government  business  practices, 
which  rely  on  cost-reimbursable  contracts.  However,  it  can  become 
more  completely  integrated  into  a  singular  industrial  base  by  acting 
like  its  partners  and  using  a  contract  vehicle  called  firm  fixed  price 
(FFP).  The  definition  of  an  FFP  is  derived  from  the  Federal  Acquisition 
Regulation  (FAR,  2005): 

A  firm-fixed-price  contract  provides  for  a  price  that  is  not 
subject  to  any  adjustment  based  on  the  contractor’s  cost 
experience  in  performing  the  contract. ..It  provides  maximum 
incentive  for  the  contractor  to  control  costs  and  perform 
effectively  and  imposes  a  minimum  administrative  burden 
upon  the  contracting  parties.  The  contracting  officer  may  use 
a  firm-fixed-price  contract  in  conjunction  with  an  award-fee 
incentive  (see  16.404)  and  performance  or  delivery  incentives 
(see  16.402-2  and  16.402-3)  when  the  award  fee  or  incentive 
is  based  solely  on  factors  other  than  cost.  The  contract  type 
remains  firm-fixed-price  when  used  with  these  incentives. 

This  contract  type  can  be  a  contentious  issue.  Although  an  FFP 
may  align  depot  incentives  with  performance  objectives,  similar  to 
FFP  use  with  a  contractor,  an  FFP  contract  with  the  depot  would 
shift  a  burden  of  risk  to  the  depots.  Historically,  this  is  not  an  area  of 
risk  that  the  depots  have  had  to  assume.  On  the  other  hand,  an  FFP 
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could  open  up  opportunities  for  the  government  operation  to  lever¬ 
age  incentives  and  reduce  costs. 

According  to  the  Office  of  the  Naval  Air  Systems  Command 
(NAVAIR)  Comptroller,  "Sales  of  DoD  goods  and  services  to  private- 
sector  entities  on  a  fixed-price  basis  are  authorized  when  the  work 
is  well  defined  and  there  is  a  reasonable  basis  upon  which  to  predict 
costs”  (DoD,  2005).  This  is  analogous  with  private-sector  practices, 
improves  the  ability  of  private-sector  partners  to  predict  production 
costs,  and  serves  to  constrain  unit  cost  by  more  fully  utilizing  the 
production  capacity  of  DoD  maintenance  depots.  Cost-reimbursable 
pricing  is  appropriate  when  future  production  costs  cannot  be  rea¬ 
sonably  predicted  (Camacho,  2008). 

Through  participation  in  an  FFP,  the  government  post-production 
support  organizations  would  create  an  opportunity  to  positively 
influence  Net  Operating  Result  over  the  life  of  the  contract;  this  is 
the  other  side  of  the  risk  coin.  However,  if  success  under  an  FFP 
occurred,  the  FRCSE  would  “earn”  funds  to  invest  in  capital  equip¬ 
ment:  variances  can  be  reinvested  in  the  depot.  This  could  create  a 
funding  source  to  facilitate  earlier  standup  of  depot  capabilities  and 
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Figure  3.  A  U.S.  Air  Force  E-8C  Joint  Surveillance  Target  Attack  Radar  System  (Joint 
STARS)  aircraft  assigned  to  the  128th  Expeditionary  Air  Command  and  Control 
Squadron.  The  Joint  STARS  is  a  battle  management  and  command  and  control  aircraft 
that  tracks  ground  vehicles  and  some  aircraft,  collects  imagery,  and  relays  tactical 
pictures  to  ground  and  air  theater  commanders.  U.S.  Air  Force  photo  by  SSgt  Aaron 
D.  Allmon  II. 
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facilitate  the  establishment  of  a  single  authoritative  source  of  depot 
repair  for  the  programs. 

Finally,  no  single  business  office  spans  the  Navy  post-production 
support  capabilities,  or  even  the  depots  themselves.  Each  depot 
maintains  its  own  business  office,  using  policies  and  practices  in  line 
with  the  commander’s  intent  for  that  installation.  While  this  maxi¬ 
mizes  flexibility  at  the  operating  level,  it  complicates  efforts  to  deliver 
needed  standardization  and  reforms. 

The  existence  of  open  issues  serves  to  illustrate  that  more  effec¬ 
tive  collaboration  across  the  industrial  base,  spanning  government 
and  industry  resources,  is  a  continuing  work  in  progress.  Partnership 
at  the  strategic  level  is  possible,  and  Jacksonville,  just  like  Huntsville, 
is  building  an  organizational  climate  that  drives  success.  And,  as  we 
continue  to  see  across  our  case  studies,  more  effective  government 
participation  is  mutually  beneficial  for  the  warfighter,  the  organic 
structure,  and  the  taxpayer. 


Case  Study  No.  3:  The  Joint  STARS  Contract— A 
Decade  of  Success 

The  E-8  Joint  Surveillance  Target  Attack  Radar  System  (Joint 
STARS)  is  a  U.S.  Air  Force  airborne  battle  management  command 
and  control,  intelligence,  surveillance,  and  reconnaissance  platform 
(Figure  3)  that  conducts  ground  surveillance  to  develop  an  under¬ 
standing  of  the  enemy  situation,  and  supports  attack  operations  and 
targeting  that  contribute  to  the  delay,  disruption,  and  destruction  of 
enemy  forces. 

Product  support  is  provided  through  a  Total  Systems  Support 
Responsibility  (TSSR)  contract,  with  Northrop  Grumman  Corpora¬ 
tion  designated  as  the  PSI.  From  its  inception,  the  Joint  STARS  TSSR 
contract— first  awarded  September  15,  2000— has  been  recognized 
as  a  pathfinder  in  the  Air  Force.  The  Under  Secretary  of  Defense 
for  Acquisition,  Technology,  and  Logistics  selected  the  Joint  STARS 
Future  Support  Team  to  receive  the  David  Packard  Excellence  in 
Acquisition  Award.  At  the  time,  a  Defense  Contract  Management 
Agency  spokesman  said,  “This  innovation  sets  a  benchmark  for 
partnering  with  industry  and  leverages  that  relationship  to  increase 
weapons  system  availability  while  reducing  operating  costs.” 

The  Joint  STARS  TSSR  Program  Management  Team,  located  at 
the  Warner  Robins  Air  Logistics  Center  (WR-ALC),  provides  program 
oversight.  Northrop  Grumman  has  the  responsibility,  authority,  and 
accountability  for  the  majority  of  day-to-day  sustainment.  Specifi¬ 
cally,  Northrop  Grumman  is  fully  accountable  for  OEM  and  vendor 


Performance- Based  Life  Cycle  Product  Support  Strategies 
Enablers  for  More  Effective  Government  Participation 


tasks,  depot  performance  under  a  workshare  agreement,  and  man¬ 
agement  of  platform-unique  items.  The  government  manages  and 
executes  product  support  for  the  engine,  common  repairables,  com¬ 
mon  consumables,  and  common  support  equipment. 

Depot  and  depot-level  repair  work  is  executed  via  partnership 
between  the  government  depot  at  WR-ALC  and  Northrop  Grum¬ 
man.  Northrop  Grumman  performs  periodic  depot  maintenance  and 
modifications  on  Joint  STARS  and  all  software  integration.  Some 
software  support  is  performed  at  WR-ALC  under  partnership,  and 
Northrop  Grumman  executes  some  software  support.  Likewise,  some 
prime  mission  equipment  repair  is  performed  by  WR-ALC  under 
partnership,  while  other  prime  mission  equipment  repair  is  handled 
by  Northrop  Grumman.  The  engine  is  managed  and  maintained  at 
the  Oklahoma  City  Air  Logistics  Center. 

Rather  than  the  traditional  approach  to  TSSR,  which  tended  to  be 
a  platform-level  agreement  with  broad  scope  provided  to  the  PSI,  the 
government  program  structure  maintains  an  active  and  visible  role 
in  directing,  managing,  and  executing  the  product  support  strategy, 
while  at  the  same  time  empowering  a  commercial  entity  as  the  PSI. 
It  is  an  integrated  approach,  bringing  together  core  competencies 
across  the  breadth  of  the  industrial  base,  and  tailoring  the  portfolio 
to  meet  the  requirements  of  this  strategic  weapon  system. 

The  net  effect  is  an  active  and  valuable  role  for  the  depots. 

Enduring  Performance 

The  Joint  STARS  is  a  complex  suite  of  technology  riding  on  an 
antiquated  airframe,  the  Boeing  707.  Yet  despite  these  challenges,  the 
integrated  performance  of  the  PSI  has  consistently  met  all  require¬ 
ments,  even  though,  for  example,  the  PSI  has  no  direct  authority  over 
depot  support.  Northrop  Grumman,  over  the  past  6  years,  in  every 
6-month  award  period  has  always  earned  within  a  few  percentage 
points  of  the  maximum  award  fee  available  under  the  contract.  And,  in 
an  attempt  to  address  a  common  criticism  of  award-fee  approaches, 
the  program  has  defined  objective  criteria  for  setting  an  award  fee. 
Since  80  percent  of  the  award-fee  recommendation  is  driven  by 
specific  and  defined  performance  outcomes,  clearly,  the  Joint  STARS 
platform  is  performing  to  expectations. 

A  defined  protocol  for  making  award-term  decisions  also  exists. 
Initially  awarded  with  a  6-year  base  period,  the  Joint  STARS  con¬ 
tract  was  configured  to  allow  up  to  an  additional  2  years  of  contract 
performance,  based  solely  on  performance,  during  each  year.  As  of 
the  end  of  2009,  Northrop  Grumman  had  already  earned  contract 
extensions  through  2017. 
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Enabling  Best  Practices 

The  complexity  of  integrating  a  product  support  strategy  as 
complex  as  Joint  STARS  into  a  functioning,  integrated  whole  is 
considerable.  To  keep  the  program  aligned,  the  team  has  brought 
together  a  tapestry  of  interwoven  checks,  balances,  and  incentives 
to  drive  desired  outcomes.  Although  each  of  these  approaches  is  a 
best  practice,  the  integration  of  all  of  these  practices  into  a  single 
strategy  is  truly  best  in  class. 

By  any  benchmark  in  the  world  of  product  support,  a  base  period 
of  6  years  is  long.  To  provide  a  secure  umbrella  under  which  the  busi¬ 
ness  partnership  could  flourish,  the  Air  Force  elected  to  look  to  a 
longer  horizon.  Instead  of  only  rewarding  a  contractor  for  excellent 
performance  with  additional  award  fee,  it  rewards  the  contractor  by 
extending  the  contract  period  of  performance  without  a  new  com¬ 
petition.  Under  an  award-term  incentive,  the  government  monitors 
and  evaluates  the  contractor’s  performance,  and  if  specific  criteria 
are  met,  additional  contract  length  is  automatically  awarded. 

For  Joint  STARS,  the  base  period,  coupled  with  the  opportunity 
for  the  contractor  to  earn  an  award-term  incentive,  leads  to  a  total 
potential  opportunity  of  22  years  to  perform.  The  Joint  STARS  PSI  has 
a  powerful  incentive  to  both  perform  and  make  life  cycle  decisions 
across  a  long  horizon.  This  is  yet  another  example  of  a  successful  best 
practice  that  could  be  more  generally  used;  however,  not  all  of  the 
Services  choose  to  recognize  award-term  contracts  as  an  available 
enabler  to  drive  performance. 

It  is  possible,  during  1  year  of  performance,  to  earn  an  additional 
2  years  of  term.  However,  the  award-term  provisions  cut  both  ways;  if 
the  PSI  performs  poorly,  it  can  lose  performance  period.  This  clearly 
encourages  consistent  and  reliable  performance. 

In  many  circles,  private  industry  is  reluctant  to  embrace  workshare 
arrangements  with  depot  resources,  because  industry  has  neither 
contractual  control  over  the  resource,  nor  the  opportunity  to  earn 
revenue/profit  on  the  work  at  the  depots.  At  WR-ALC,  a  workshare 
arrangement  is  in  place,  but  a  business  model  has  also  been  put  in 
place  to  incentivize  the  PSI  to  influence,  and  hopefully  drive,  perfor¬ 
mance  at  the  depots.  Simply  put,  the  PSI  can  earn  award  fee  based 
on  depot  performance.  This  simple  step  makes  the  PSI  a  stakeholder, 
deeply  invested  in  making  the  depots  successful. 

Deep  implications  are  also  inherent  to  the  award-fee  approach 
with  Joint  STARS.  Typically,  award  fee  is  distributed  based  on  subjec¬ 
tive  judgment.  Instead  of  the  conventional  approach,  the  program 
has  defined  objective  criteria  for  determining  award  fee.  By  defin¬ 
ing  specific  and  objective  measures  and  using  those  to  determine 
the  distribution  of  award  fee,  the  Air  Force  has  driven  alignment  to 
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specific  outcome  criteria  across  the  program.  This  is  a  key  principle 
of  performance-based  product  support. 

Tools  have  been  developed  to  allow  the  PSI  to  augment  gov¬ 
ernment  organizational  performance  when  necessary.  The  PSI  is 
authorized,  when  requested  to  do  so  by  the  government,  to  provide 
common  item(s)  when  the  government  item  manager’s  estimated 
delivery  date  does  not  meet  the  warfighters’  need  date.  The  PSI  is 
also  authorized  to  handle  surge  workload  and  shortfalls  when  the 
capacities  at  WR-ALC  handling  repair  of  mission  systems  are  unable 
to  meet  the  requirement. 

Open  Issues 

One  of  the  most  difficult  issues  in  establishing  long-term,  perfor¬ 
mance-based  contracts  is  the  establishment  of  objective  performance 
outcome  measures  that  remain  relevant,  challenging,  and  attain¬ 
able  over  the  life  cycle.  Today,  almost  10  years  into  the  Joint  STARS 
TSSR,  the  PSI  and  the  Air  Force  are  revisiting  the  measures  used  to 
develop  award-fee  recommendations.  Past  attempts  to  modify  the 
targets  have  stalled  because  the  targets  are  contractual  terms,  and 
any  modification  requires  mutual  consent. 

The  grinding  requirements  of  ongoing  operations  have  caused  a 
shift  in  perspective.  Today,  warfighters  express  a  greater  interest  in 
aircraft  availability  and  sortie  effectiveness.  Consequently,  the  pro¬ 
gram  team  is  working  to  rearrange  the  weights  of  certain  governing 
metrics.  Today,  17  metrics  roll  up  into  a  final  weighted  score.  Ideally, 
one  of  these  metrics— Depot  Possessed  Aircraft— can  be  moved  from 
12  percent  of  the  total  to  20  percent;  and  Introductory  Flight  Train¬ 
ing  sortie  effectiveness  can  be  increased  from  a  mere  2  percent  to 
10  percent.  This  10-percentage-point  weight  shift  would  come  by 
reducing  the  relative  weight  of  cost  measures. 

The  determination  of  what  the  right  weights  should  be  is  a  discus¬ 
sion  best  left  to  the  team  most  familiar  with  the  weapon  system;  and 
the  number  of  measures  tracked  as  top-level  outcomes  is  open  to 
debate.  This  process,  however,  highlights  the  need  to  build  reset  and 
calibration  mechanisms  into  measurement  schemes  to  allow  outcome 
definitions  over  time. 

As  with  the  previous  cases,  Joint  STARS  has  opportunities  to 
continue  with  its  improvement  journey  and  deliver  more  effective 
performance.  That  said,  the  innovations  we  see  at  WR-ALC,  proven 
over  the  last  decade,  demonstrate  that  enablers  are  available  to  drive 
best-value  participation  across  a  breadth  of  government  resources. 
And,  once  again,  we  see  that  more  effective  government  participation 
promotes  increased  synergy  and  collaboration  for  the  warfighter,  the 
organic  structure,  and  the  taxpayer. 
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Figure  4.  An  HH-60G-Pave  Hawk  helicopter  from  the  33rd  Rescue  Squadron  (RQS) 
receives  fuel  from  a  KC-130J  during  a  3-day  intensive  air  refueling  course  at  Kadena 
Air  Base,  Japan.  The  KC-130J,  which  performs  air-refueling  missions,  is  a  specialized 
version  of  the  C-130J,  a  medium-range,  tactical  aircraft  and  the  newest  upgrade  to  the 
C-130  fleet.  U.S.  Air  Force  photo  by  SSgt  Chrissy  Best. 


Case  Study  No.  4:  The  Upstarts— Naval  Surface 
Warfare  Center,  Crane  Division 

The  C-130J  is  a  modification  of  the  C-130H,  undertaken  by 
Lockheed  Martin  Aeronautics  Corporation  (LMAC)  as  a  private  ven¬ 
ture,  with  intended  sales  to  the  United  States  and  various  foreign 
markets.  The  C-130J  aircraft  is  a  medium-range,  tactical  aircraft 
and  is  the  newest  upgrade  to  the  C-130  fleet.  Specialized  versions 
of  the  aircraft  include  the  C-130J  Stretch,  which  has  an  increased 
cargo  floor  length  of  15  feet;  the  WC-130J,  which  performs  weather 
reconnaissance  missions;  the  EC-130J,  which  performs  electronic 
warfare  missions;  the  KC-130J,  which  performs  air-refueling  mis¬ 
sions  (Figure  4);  and  the  HC-130J,  which  performs  search  and 
rescue  missions. 

Currently,  the  U.S.  Government  operates  approximately  100  air¬ 
frames,  with  65  in  the  U.S.  Air  Force,  29  in  the  U.S.  Marine  Corps,  and 
6  in  the  U.S.  Coast  Guard;  another  60  are  owned  by  foreign  govern¬ 
ments.  Historical  practice  would  suggest  that  since  the  C-130J  was 
built  using  private  investment,  the  military  would  rely  on  a  system- 
level,  performance-based  product  support  acquisition  strategy, 
with  the  OEM  as  either  the  integrator  or  playing  an  active  role  in  the 
integration.  That  is  not  the  case  in  the  Navy. 

The  NAVAIR-NSWC  Crane  Partnership 

The  Air  Force  supports  the  C-130J  under  a  long-term,  PBL  part¬ 
nership  among  LMAC,  the  C-130  Program  Office,  and  the  330th  Air 
Combat  Support  Group  at  WR-ALC.  Initially,  the  NAVAIR  followed 
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the  U.S.  Air  Force  product  support  strategy  and  relied  on  LMAC 
as  the  source  of  supply  for  KC-130J  platform-unique  components. 
However,  as  operational  requirements  and  ongoing  commitments 
grew  without  proportionate  additions  to  budgets,  the  Navy  found 
itself  under  financial  pressure.  Seeking  alternatives,  and  unable  to 
afford  the  pricing  available  through  LMAC,  the  NAVAIR  program 
office  opened  up  a  dialogue  with  the  Naval  Surface  Warfare  Center 
(NSWC),  Crane  Division. 

In  collaboration  with  the  program  office,  NSWC  Crane  began 
seeking  alternative  repair  item  sourcing  strategies  for  the  KC-130J. 
Since  the  C-130J  is  a  complex  weapon  system,  subcontractors  pro¬ 
duce  many  items  on  behalf  of  LMAC.  Additionally,  the  government 
owned  technical  data  for  many  of  the  components.  The  solution 
Crane  offered  was  simple:  It  would  replace  LMAC  as  a  supply  chain 
integrator  at  the  component  level  for  the  program  office,  and  reach 
out  directly  to  the  supplier  community.  This  arrangement  offered 
the  additional  benefit  of  swift  implementation  without  the  need  for  a 
Business  Case  Analysis:  NSWC  Crane  is  within  the  same  Service  and 
can  readily  accept  Military  Interdepartmental  Purchase  Requests. 

According  to  the  program  office,  NSWC  Crane  has  been  extremely 
successful  as  an  agent,  driving  dramatic  cost  reductions  in  costs  per 
flight  hour,  and  in  many  cases  obtaining  warranty  coverage  superior 
to  that  available  from  LMAC.  Further,  NSWC  Crane  is  behaving  entre¬ 
preneurial^  and,  in  conjunction  with  the  program  office,  has  identified 
a  way  to  apply  the  next-generation  business  model  described  in  the 
Product  Support  Assessment  (DoD,  2009)  to  its  advantage. 

As  reported  by  the  NSWC  Public  Affairs  Office,  logistically  reen¬ 
gineering  the  sustainment  program  and  re-baselining  “by  the  flight 
hour”  has  been  successful  (Camacho,  2008).  NSWC  Crane  receives  a 
fixed  rate  for  each  KC-130J  flight  hour  flown  and  promises  a  specific 
minimum  level  of  performance.  The  project  team  employs  continu¬ 
ous  improvement  Lean  tools  in  keeping  with  NSWC  Crane's  continual 
efforts  to  provide  timely,  affordable,  and  quality  solutions  to  the 
warfighter.  This  approach  helped  increase  the  desired  efficiencies  that 
ultimately  benefited  flight-hour  costing  and  mission  capability.  The 
minimum  level  of  performance  was  set  at  85  percent  mission  capa¬ 
bility  due  to  supply  issues,  but  successfully  executed  in  excess  of  95 
percent  since  support  moved  to  NSWC  Crane.  In  2007,  NSWC  Crane's 
role  in  KC-130J  sustainment  had  saved  the  government  $42  million 
by  reducing  the  cost  per  flight  hour  by  nearly  75  percent  from  2005 
to  2007  (Camacho,  2008).  More  recently,  according  to  PMA-207’s 
APML,  the  relationship  with  NSWC  Crane  has  yielded  more  reduc¬ 
tions  in  operating  costs.  If  NAVAIR  had  stayed  with  LMAC,  estimates 
of  the  current  cost  are  more  than  $1,000  per  flight  hour.  At  times,  the 
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KC-130J  has  operated  under  $300  per  flight  hour  for  unique  repair 
of  KC-130J  repairables. 

The  government  post-production  support  element  at  NSWC 
Crane  has  leveraged  its  skill  and  operates  as  a  viable  competitor  to 
the  commercial  OEM  as  a  PSI  on  an  FFP  basis. 

Open  Issues 

While  the  strategy  employed  on  the  KC-130J  is  innovative  and 
successful,  there  are  risks.  Bypassing  LMAC  moves  NAVAIR  and 
NSWC  Crane’s  PBL  out  from  under  the  umbrella  of  LMAC.  To  mitigate, 
NAVAIR  contracts  for  technical  support  from  LMAC  through  another 
arrangement  that  is  managed  as  a  part  of  the  program  portfolio.  The 
program  office  has  elected  to  retain  more  responsibility— and  more 
risk— by  accepting  a  more  active  and  central  role  in  the  execution  of 
the  support  strategy. 

To  illustrate  the  potential  risks  of  the  approach,  consider  the  life 
cycle.  The  KC-130J  is  a  maturing  platform,  and  obsolescence  chal¬ 
lenges  as  well  as  diminishing  manufacturing  sources  of  supply  can 
be  anticipated.  Will  the  program  office  and  NSWC  Crane  be  able  to 
manage  transitions  as  effectively  as  LMAC?  Or  would  NAVAIR  be 
better  off  by  involving  the  OEM  more  directly  in  the  PBL  strategy 
through  some  sort  of  integrated  accountability  for  performance 
and  outcomes,  instead  of  acquiring  technical  support  in  a  fee-for- 
service  arrangement?  There  are  trade-offs,  and  costs  to  date  have 
clearly  been  positively  impacted  by  the  arrangement,  but  as  the 
platform  matures,  a  strategy  review  may  be  appropriate  to  ensure 
continuing  success. 

As  NAVAIR  and  the  U.S.  Air  Force  have  charted  independent 
courses,  they  have  disaggregated  the  support  strategy  for  the  plat¬ 
form  itself.  The  U.S.  Air  Force  maintains  a  separate  program  office 
at  WR-ALC,  with  its  own  strategy  and  portfolio  of  contracts.  Against 
the  imperatives  of  the  individual  Services,  reasonable  managers  have 
made  reasonable  decisions.  However,  opportunities  for  cross-Service 
standardization  and  cross-pollination  may  exist. 

All  things  considered,  in  a  climate  of  increasing  financial  challenges 
the  program  office  has  answered  the  call  for  innovation  and  creativity. 
More  effective  government  participation  is  possible,  and  it  is  mutually 
beneficial  for  the  warfighter,  the  organic  structure,  and  the  taxpayer. 

Findings  and  Recommendations 

The  next-generation  product  support  strategy  will  not  deliver 
unless  the  whole  community,  including  both  government  and  com¬ 
mercial  industry,  is  able  to  make  the  necessary  changes  in  behaviors, 
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organizations,  and  business  processes.  The  necessary  changes,  as  we 
have  shown  in  the  case  studies,  include  the  following  10  strategies: 


STRATEGY  NO.  1 

Integrating  of  government  post-production  support  capabilities 
as  best-value  partners  into  a  unified  industrial  base. 


STRATEGY  NO.  2 

Creating  the  correct  blend  of  government  and  industry  partner¬ 
ship  based  on  best  value  capabilities,  not  statutory  entitlement. 


STRATEGY  NO.  3 

Defining  the  PSI  role  based  upon  program  requirements  and  not 
dogma. 


STRATEGY  NO.  4 

Leveraging  incentive  strategies  in  the  government-owned  and 
-managed  resources  to  drive  down  life  cycle  cost. 


STRATEGY  NO.  S 

Capitalizing  on  the  government  post-production  support  organi¬ 
zations’  ability  to  perform  at  affordable  prices. 


STRATEGY  NO.  6 

Creating  a  culture  of  high-performing,  innovation-driven  govern¬ 
ment-industry  teams. 


STRATEGY  NO.  7 

Sharing  vision  and  tying  that  vision  to  objectives,  metrics,  and 
incentives. 


STRATEGY  NO.  8 

Understanding  all  stakeholder  interests  and  striving  for  win-win. 
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STRATEGY  NO.  9 

Seeking  common  ground,  with  a  shared  view  of  a  common  end 
customer;  what  unites  government  and  industry  should  be  stronger 
than  what  divides. 


STRATEGY  NO.  10 

Understanding  incentives  (FFP,  award  term,  incentive  fee,  etc.). 

What  we  have  seen  in  a  crosscutting  sample  of  government  post¬ 
production  support  organizations’  participation  in  performance-based 
life  cycle  product  support  strategies  is  that  government  organizations 
can  effectively  and  aggressively  participate  and  compete. 

We  have  included  examples  of  government  post-production  sup¬ 
port  organizations  from  each  of  the  Services,  and  have  taken  care 
to  include  programs  from  a  spectrum  of  commercial  companies, 
including  General  Electric,  Raytheon,  Northrop  Grumman,  and  Lock¬ 
heed  Martin.  The  examples  presented  demonstrate  that,  regardless 
of  the  perceived  obstacles,  determined  and  motivated  government 
post-production  support  organizations  can  identify  opportunities 
and  compete  effectively  and  successfully.  What  we  are  now  seeing 
in  the  government  is  the  emergence  of  competitive  organizations, 
fully  capable  of  participation,  not  as  a  matter  of  entitlement,  but  as 
a  matter  of  competence. 

Adoption  of  partnership  approaches  on  a  broader  scope  neces¬ 
sarily  provides  impetus  to  the  cross-fertilization  of  best  practices 
between  industry  and  the  government  post-production  support 
base.  At  the  same  time,  there  exists  considerable  core  competency 
in  the  government  community,  particularly  in  human  capital  and 
infrastructure,  which  means  that  there  should  be  cross-fertilization 
from  the  government  post-production  support  base  to  industry.  In 
the  General  Electric  example,  we  have  seen  the  Jacksonville  FRCSE 
moving  into  a  new  line  of  business— OEM— because  General  Electric 
views  the  government  capabilities  as  more  cost  effective  than  its  own. 

The  Product  Support  Assessment  (DoD,  2009)  describes  a  vision¬ 
ary  agenda  for  structural  change  to  facilitate  a  more  integrated 
industrial  base.  In  fact,  it  recommends  that  DoD  “Propose  modifi¬ 
cations  to  Title  10  to  enable  maximum  implementation  of  industrial 
integration.”  The  report  then  elaborates: 

A  rethinking  of  the  nature  of  partnership  includes  statutory 
requirements  and  issues  which  may  impede  effective  and 


Performance- Based  Life  Cycle  Product  Support  Strategies 
Enablers  for  More  Effective  Government  Participation 


affordable  implementation  of  a  warfighter-based  product 
support  strategy.  A  more  consistent  approach  to  financial 
rules  and  incentives,  putting  government  and  commercial 
organizations  on  equal  footing,  will  inevitably  lead  to  results 
that  are  more  predictable.  Revised  or  new  statutory  require¬ 
ments  should  do  three  things: 

1.  Propose  a  strategy  for  enabling,  requiring,  and  monitor¬ 
ing  the  ability  of  the  Department  of  Defense  supply  chain 
offices  and  industrial  activities  to  produce  performance- 
driven  outcomes  and  meet  materiel  readiness  goals  with 
respect  to  availability,  reliability,  total  ownership  cost,  and 
repair  cycle  time. 

2.  Enable  industry  investment  in  DoD’s  industrial  and  other 
product  support  activities  by  submitting  a  legislative 
change  to  modify  the  government  ownership  requirement 
of  depot  and  other  support  equipment  and  facilities  used 
in  support  of  core  capabilities. 

3.  Establish  reporting  constructs  to  stimulate  financial  and 
cost  reporting  equivalency  (i.e.,  comparable)  between 
industry  and  the  government,  and  require  cost  transpar¬ 
ency  to  the  greatest  extent  possible  while  respecting  the 
need  to  protect  competition-sensitive  information,  (p.  45) 

As  we  have  illustrated  through  the  case  studies  in  this  article,  an 
active  and  vibrant  community  across  the  defense  industrial  base  is 
already  bringing  the  vision  of  the  Product  Support  Assessment  (DoD, 
2009)  to  life.  The  initiatives  proposed  in  the  report,  as  time  has  proven, 
eventually  served  as  a  catalyst  to  the  community’s  current  success. 

As  previously  cited  in  the  Jacksonville  case  study,  the  report  also 
recommends,  “Establish  policy  and  training  to  expand  partnering 
‘beyond  maintenance,’  drive  standardization  across  Services,  and 
promote  proactive  establishment  of  single-source  repair  capability.” 
As  we  have  seen  in  this  report,  ample  opportunity  and  proven  best 
practices  are  available  to  fuel  this  effort  in  the  government  post¬ 
production  support  structure. 

How  to  interpret  and  apply  the  examples  presented  in  this  report 
is  subjective,  but  within  the  context  of  establishing  policy  and  training, 
driving  standardization,  and  promoting  single-source  repair  capabil¬ 
ity,  specific  actions  are  possible. 
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ACTION  NO.  1 

The  Defense  Acquisition  University  (DAU)  should  train  leadership 
levels  within  the  government  post-production  support  organizations 
on  how  to  apply  the  critical  success  factors  uncovered  at  Huntsville 
and  demonstrate  their  linkage  to  PBL.  Leverage  academic  and  practi¬ 
cal  expertise  to  provide  managers,  senior  managers,  and  executives 
with  an  understanding  and  ability  to  create  a  team  climate  for  inno¬ 
vation. 


ACTION  NO.  2 

Highlight  the  ability  of  the  government  post-production  support 
organizations  to  make  use  of  incentives  paid  to  hourly  workers,  and 
demonstrate  how  to  align  that  with  outcome-based  product  support 
strategies. 


ACTION  NO.  3 

Make  visible  the  proven  utility  and  legality  of  FFP  contract 
approaches  at  the  depots. 


ACTION  NO.  4 

Train  the  government  post-production  support  organizations  in 
identifying  their  core  competencies,  and  establish  business  plans  to 
grow,  manage,  and  market  these  capabilities  across  programs  and 
Services. 


ACTION  NO.  S 

Develop  case  studies  on  taking  a  portfolio  approach  to  depot 
standup,  and  build  single-source  repair  capability  from  the  ground 
up,  incrementally. 


ACTION  NO.  6 

Promote  the  long-term  success  of  the  Joint  STARS  program  to 
demonstrate  that  hybrid  approaches  utilizing  long-term  contracts 
can  be  successful. 
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ACTION  NO.  7 

Provide  guidelines  and  training  to  appropriate  government  post¬ 
production  support  organizations  on  the  business  opportunities 
available  if  core  capabilities  as  a  PSI  for  supply  chain  integration  are 
developed  and  marketed. 


ACTION  NO.  8 

Support  the  development  of  training  materials  and  case  studies  at 
DAU  based  on  the  government  post-production  support  organization 
successes  documented  in  this  report. 


ACTION  NO.  9 

Create  virtual  business  offices  for  each  Service  and  a  mechanism 
to  promote  standardization  while  leaving  the  resources  resident  in 
the  individual  commands. 


ACTION  NO.  10 

Continue  to  drive  for  the  adoption  of  performance-based  product 
support  across  the  enterprise,  and  use  the  examples  in  the  report 
to  demonstrate  the  opportunity  that  this  approach  provides  for  the 
government  post-production  support  base. 

Success  enablers  are  abundant  for  government  post-production 
support  organizations  to  participate  in  performance-based  life  cycle 
product  support  strategies.  It’s  time  to  spread  the  knowledge. 
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A  NEW  PROCESS 

FOR  THE  ACCELERATION 
TEST  AND  EVALUATION 
OF  AEROMEDICAL 
EQUIPMENT  FOR 
U.S.  AIR  FORCE 
SAFE-TO-FLY 
CERTIFICATION 

i  Ismail  Cicek  and  Capt  Gary  S.  Beisner,  USAF 

Aeromedical  flight  equipment  must  meet  airworthiness  criteria 
according  to  Department  of  Defense  Handbook  MIL-HDBK-516, 
Airworthiness  Certification  Criteria,  MIL-STD-810G,  and  MIL-STD- 
1791,  which  requires  restraint  of  any  item  that  may  potentially 
cause  injury  to  personnel  during  emergency  landings,  an  over¬ 
water  ditching,  or  crash  loads.  Several  government  standards 
provide  adequate  descriptions  of  acceleration  test  methods; 
however,  none  formally  documents  a  non-destructive  test 
method  to  qualify  equipment  as  safe-to-fly  (STF).  Using  the 
USAF  fixed-wing  aircraft  STF  test  criteria,  this  article  presents 
a  structured  process  developed  by  the  Aeromedical  Test 
Branch,  77th  Aeronautical  Systems  Group,  to  assess  equip¬ 
ment  as  STF.  Further,  it  demonstrates  the  application  of  this 
process  to  meet  the  acceleration  requirements  for  aeromedical 
evacuation  equipment. 


Keywords:  Acceleration  Test,  Aeromedical 
Equipment,  Safe-to-Fly  Process,  Test  and  Evaluation, 
Aeromedical  Evacuation,  USAF  Aircraft 


image  designed  by  Miracle  Riese  » 


48  6 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


Generally  speaking,  medical  devices  are  designed  to  function  in 
environmentally  controlled  locations,  such  as  stationary  hospitals, 
and  not  within  the  harsh,  dynamic  aircraft  environment.  Yet,  the  same 
medical  devices  used  to  care  for  patients  in  a  hospital  environment 
are  often  the  most  capable  devices  for  patient  care  during  transport 
from  one  facility  to  another.  These  missions  are  called  aeromedical 
evacuations  (AE)  missions,  and  they  provide  life-sustaining  care  for 
a  vast  array  of  patients.  However,  because  the  devices  are  designed 
for  a  controlled  environment,  concerns  they  may  adversely  affect 
the  operation  of  aircraft  systems  must  be  addressed.  Conversely,  the 
aircraft  may  adversely  affect  the  proper  operation  and  efficacy  of  the 
medical  equipment. 


USAF  STF  Test  Process  for  AE  Equipment 

Failure  of  medical  devices  during  in-flight  medical  care  may  result 
in  exposing  patients  and  aircrew  to  hazardous  situations.  All  medical 
equipment  identified  for  use  on  U.S.  Air  Force  AE  fixed-wing  aircraft 
must  undergo  a  STF  test  process  in  accordance  with  Section  2. 5.1. 7  of 
Air  Force  Instruction  11-202  (Department  of  Air  Force,  2006),  before 
the  STF  certification  can  be  issued  by  the  authorizing  aircraft  system 
organizations.  Military  standards,  civilian  regulations,  and  professional 
experience  and  expertise  are  all  part  of  the  STF  evaluation  package. 

A  typical  STF  evaluation  features  three  phases. 

Phase  I:  Baseline  Assessment 

The  purpose  of  the  baseline  assessment  is  to  verify  that  the 
equipment  under  test  (EUT)  operates  in  accordance  with  the  man¬ 
ufacturer’s  specifications  and  the  operator’s  manual.  The  EUT  is 
evaluated  for  adherence  to  optimum  human  factors  referenced  in 
MIL-STD-1472F  (Department  of  Defense,  1999)  and  basic  electrical 
safety  requirements.  The  test  team  becomes  familiar  with  the  equip¬ 
ment  to  select  the  appropriate  tests  based  on  the  U.S.  Air  Force  AE 
equipment  test  requirements.  From  there,  the  team  identifies  the 
tie-down  configuration,  aircraft  interfaces,  and  operational  use  of 
the  equipment  during  the  baseline  assessment.  The  test  plan  is  then 
developed  and  submitted  to  the  aircraft  system  organizations  for 
review  prior  to  starting  the  laboratory  tests. 

Phase  II:  Laboratory  Tests 

The  purpose  of  the  laboratory  testing  phase  is  to  simulate  the 
operational  in-flight  environment  through  testing,  which  is  modeled 
after  a  series  of  worst-case  event  scenarios,  such  as  a  rapid  decom- 
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pression  event  or  other  aircraft  incidences  or  mishaps.  Military  and 
industrial  standards  are  used  as  guidance  to  select  the  tests  and 
establish  the  test  criteria.  Typical  laboratory  tests  include  vibra¬ 
tions,  electromagnetic  interference  (EMI),  hot  and  cold  temperature 
extremes  for  operational  use  and  storage,  humidity,  explosive  atmo¬ 
sphere,  altitude,  rapid  decompression,  and  acceleration. 

Prior  to  2006,  specific  types  of  aircraft  were  dedicated  almost 
exclusively  to  AE  missions.  The  use  of  medical  devices  during  flight 
was  a  routine  part  of  the  daily  mission,  and  acceleration  testing 
was  not  a  solid  STF  test  requirement.  Since  then,  refinements  in  the 
employment  of  cargo  aircraft  have  enabled  a  broader  array  of  assets 
for  AE  and  other  transport  missions.  This  change  allows  any  available 
cargo  aircraft,  or  "opportune  aircraft,"  to  be  quickly  designated  and 
configured  as  an  AE  transport  aircraft.  While  this  fundamental  shift 
in  operations  greatly  benefited  the  overall  AE  mission,  more  exhaus¬ 
tive  testing  procedures  were  implemented  to  assess  medical  devices 
prior  to  in-flight  use  to  ensure  safety  across  the  numerous  aircraft 
fleets.  These  devices  were  now  expected  to  conform  to  typical  airlift 
standards  just  as  any  other  cargo  brought  on  board.  The  most  notable 
change  to  the  testing  procedures  was  the  addition  of  more  robust 
acceleration  testing  requirements.  After  the  AE  test  article  completes 
the  laboratory  phase,  an  In-Flight  Assessment  (IFA)  may  begin. 

Phase  III:  IFA 

The  purpose  of  conducting  an  IFA  for  AE  equipment  is  to  per¬ 
form  functional  checks  on  board  the  aircraft  during  an  aeromedical 
readiness  mission.  The  controls,  visual  and  audible  alarms,  and  display 
screen  of  the  AE  equipment  are  observed  and  evaluated  during  the 
flight.  Test  personnel  interact  with  and  solicit  feedback  from  AE  crew¬ 
members  regarding  the  device’s  form,  fit,  and  function.  These  data  are 
used  to  identify  any  remaining  issue  with  the  use  of  the  device  that 
may  not  arise  during  the  simulated  laboratory  test  scenarios.  Further, 
this  final  phase  also  assists  in  evaluating  and  solidifying  the  intended 
concept  of  operations  for  the  device. 


Acceleration  Testing 

AE  equipment  must  meet  airworthiness  criteria  according  to  MIL- 
HDBK-516B  (Department  of  Defense,  2008).  The  criteria  require  items 
that  could  cause  injury  to  personnel  during  emergency  landings, 
ditching,  and  crash  loads  to  be  restrained.  Since  aeromedical  devices 
are  not  mission-critical  equipment  and  are  typically  considered  carry- 
on  equipment,  the  main  thrust  for  acceleration  testing  hinges  on  the 
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inertial  loads  where  safety  is  paramount.  Successful  completion  of 
acceleration  testing  ensures  AE  equipment  can  sustain  acceleration 
loads  found  in  aviation  mishaps  and,  more  importantly,  ensures  the 
safety  of  the  aircraft’s  occupants.  Ultimately,  testing  is  used  to  ensure 
medical  devices  or  any  cargo  does  not  adversely  impact  any  chance 
of  survival  or  impede  or  prohibit  passengers'  egress.  Additionally, 
high  levels  of  acceleration  may  have  detrimental  effects  on  the  AE 
equipment,  leading  to  broken  fasteners,  supports,  and  mounting 
components.  Failures  such  as  these  may  result  in  insufficent  restraint 
of  the  device  or  its  components,  ultimately  allowing  it  to  become  a 
projectile  during  a  typical  crash  scenario.  Therefore,  the  equipment’s 
mounting  and/or  restraint  methods  must  be  tested  to  verify  that  they 
will  not  fail  and  subelements  can  be  properly  contained  within  the 
system  during  an  acceleration  event. 

Acceleration,  as  addressed  in  MIL-STD-810G  Method  513.6 
(Department  of  Defense,  2009),  is  a  load  factor  (inertial  load  or  "g" 
load)  that  is  applied  slowly  enough  and  held  steady  for  a  period  of 
time  such  that  the  materiel  has  sufficient  time  to  fully  distribute  the 
resulting  internal  loads  to  all  critical  joints  and  components.  The 
common  methods  used  to  expose  equipment  to  a  sustained  accel¬ 
eration  load  are  centrifuge  and  track/rocket-powered-sled  testing. 
However,  both  methods  impose  limitations  on  AE  equipment  testing. 
For  example,  the  costs  required  and  the  scheduling,  planning,  and 
coordination  phases  associated  with  the  use  of  these  types  of  test 
facilities  are  often  prohibitive.  In  some  cases,  centrifuges  and  track/ 
rocket  sleds  may  limit  the  orientations  at  which  the  test  article  can 
be  mounted  for  testing.  To  maintain  validity,  all  AE  devices  are  tested 
under  the  same  mounting  configuration  as  intended  for  operational 
use.  Finally,  due  to  the  often  expensive  and  delicate  nature  of  medical 
devices,  insufficient  inventories  often  prevent  the  use  of  these  tests 
due  to  their  somewhat  destructive  nature. 

Because  of  the  difficulties  associated  with  physical  dynamic  test¬ 
ing,  the  ATB  team  initially  turned  to  Finite  Element  Analysis  (FEA) 
as  the  method  of  choice  for  meeting  acceleration  test  requirements. 
Recent  technological  advances  in  microcomputing  and  higher  reso¬ 
lution  graphics  capabilities  allowed  complex  systems  to  be  modeled 
and  simulated  for  both  static  and  dynamic  tests. 

The  FEA  techniques  were  already  used  by  others  for  various 
aircraft  structures  and  devices.  For  example,  Foster  and  Sarwade 
(2005)  performed  an  FEA  of  a  structure  that  attached  medical 
devices  to  a  litter.  This  structure  was  later  approved  as  STF.  Continu¬ 
ing  on  the  same  theme,  Lawrence,  Fasanella,  Tabiei,  Brinkley,  and 
Shemwell  (2008)  studied  a  crash  test  dummy  model  for  NASA’s  Orion 
crew  module  landings  using  FEA.  Viisoreanu,  Rutman,  and  Cassatt 
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(1999)  reported  their  findings  for  the  analysis  of  the  aircraft  cargo 
net  barrier  using  FEA.  Furthermore,  Motevalli  and  Noureddine  (1998) 
used  an  FEA  model  of  a  fuselage  section  to  simulate  the  aircraft  cabin 
environment  in  air  turbulence.  These  and  similar  studies  demon¬ 
strated  the  successful  use  of  the  FEA  method  to  verify  requirements 
by  analysis  for  an  acceleration  test. 

Given  the  costs  associated  with  dynamic  testing,  the  ATB  origi¬ 
nally  envisioned  using  the  FEA  method  to  alleviate  budget  and 
inventory  concerns.  To  test  this  theory,  the  ATB  employed  FEA  for 
testing  various  AE  structures  to  meet  the  acceleration  requirements 
and  found  some  aspects  of  this  method  to  be  cost-  and  time-pro¬ 
hibitive.  Lessons  learned  from  these  studies  are  provided  in  the 
case-studies  section. 

The  various  types  of  analysis  and  test  methods  raise  questions 
as  to  what  the  correct  decision  process  is  for  selecting  the  most 
appropriate  method  for  STF  testing  of  AE  equipment.  The  authors 
of  this  article  describe  the  process  developed  and  employed  by  the 
ATB  for  the  acceleration  testing  of  AE  equipment  since  June  2008. 
The  ATB's  process  has  proven  to  be  well  suited  for  identifying  the 
most  appropriate  test  method— one  that  not  only  represents  the 
most  appropriate  and  effective  test  method,  but  also  minimizes  the 
use  of  available  resources.  This  process  includes  testing  both  struc¬ 
turally  simple  and  complex  equipment  and  successfully  introducing 
the  use  of  the  Equivalent  Load  Testing  (ELT)  method,  which  permits 
the  use  of  alternative  testing  approaches,  such  as  pull  testing  and 
tensile  testing. 


ATB's  Acceleration  Test  and  Evaluation  Process 

Process  Description 

An  integrated  team  approach  remains  the  cornerstone  for  the 
acceleration  test  and  evaluation  process  for  AE  STF  certifications. 
The  team  members,  each  having  different  skill  sets,  become  part  of 
an  acceleration  test  assessment  meeting  where  the  subject  test  item 
is  evaluated  against  the  acceleration  test  requirements  and  the  type 
of  test  is  identified.  The  team  also  identifies  the  intended  operational 
and  tie-down  configuration,  assesses  the  means  in  which  compo¬ 
nents  and  subcomponents  are  mounted  to  the  system,  and  all  other 
concerns  related  to  acceleration  requirements.  The  overall  process 
is  depicted  in  Figure  1. 

The  initial  task  of  the  integrated  team  is  to  evaluate  the  test 
article  for  any  inherent  safety  concerns.  For  example,  the  ATB  team 
identified  that  AE  devices  weighing  less  than  five  pounds  are  usually 
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perceived  to  pose  no  substantial  risks  due  to  acceleration;  therefore, 
a  quick  assessment  and  description  of  the  equipment  tie-down  were 
found  satisfactory.  Generally,  the  team  conducts  a  test  selection 
meeting  for  the  items  weighing  more  than  5  pounds. 

When  the  test  team  finds  product-level  tests  are  required,  the 
article  is  tested  in  a  physical  environment,  namely  sled  tracks  or  cen- 


FIGURE  1.  ACCELERATION  TEST  AND  EVALUATION  PROCESS 
DIAGRAM 
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trifuges,  or  a  model  representing  the  product  can  be  developed  and 
analyzed  using  FEA  simulation.  The  component-level  tests  refer  to 
the  tests  specific  to  a  subcomponent  or  a  structural  member  of  the 
equipment,  i.e.,  mounting  brackets,  screws,  beams,  straps,  etc.  When 
the  decision  is  a  component-level  test  type,  the  ATB  team  applies 
the  ELT  method  by  conducting  an  in-depth  evaluation  of  the  test 
article,  identifying  the  critical  areas  within  the  item,  and  noting  any 
potential  safety  concerns  within  the  environment.  The  outcome  of 
this  evaluation  is  a  list  of  tasks  that  includes  a  series  of  tests,  analysis, 
inspections,  and  evaluations.  The  component-level  test  requires  a 
final  assessment  meeting  where  the  ATB  team  analyzes  and  deems 
acceptable  the  component-level  test  results  or  determines  whether 
additional  tasks  are  required. 

Prior  to  2008,  acceleration  tests  were  typically  conducted  at 
the  product  level.  However,  the  case  studies  presented  in  this  article 
highlight  a  multitude  of  alternative  test  options  for  component-level 
testing  as  well.  When  selecting  between  product  or  component-level 
testing,  the  ATB  carefully  considers  many  different  aspects  of  the 
overall  design  of  the  equipment,  its  intended  use,  and  any  unique 
safety  concerns.  For  example,  component-level  testing  would  most 
likely  not  be  adequate  for  a  system  containing  compressed  gas  cylin¬ 
ders  because  this  form  of  testing  would  only  target  the  key  structural 
features  of  the  system,  such  as  the  handles  on  the  outer  case.  Rather, 
a  product-level  test,  such  as  a  sled  test,  would  be  more  appropriate 
as  it  would  test  the  whole  system  including  the  connections  between 
the  cylinders  and  the  system  where  dangerous  leaks  could  occur. 

The  component-level  test  was  recently  added  to  the  accelera¬ 
tion  test  process  and  has  saved  a  significant  amount  of  time  and 
money  since  2008;  therefore,  the  ATB  places  emphasis  on  the  com¬ 
ponent-level  test,  unless  the  product  must  be  tested  in  a  physical 
environment.  The  component-level  test  uses  the  ELT  method,  which 
is  detailed  in  the  following  section. 

The  ELT  Method 

The  ELT  terminology  used  in  this  article  refers  to  the  constant, 
or  approximately  constant,  loading  that  is  applied  to  the  test  item 
for  a  finite  duration.  The  magnitude,  point  of  application,  and  the 
direction  of  the  load  are  equivalent  to  the  properties  of  inertial  loads 
and  moments  generated  in  an  acceleration  event  under  the  g-levels 
shown  in  Table  1.  The  levels  represented  in  this  table  are  some  of  the 
common  test  requirements  for  AE  equipment  as  outlined  by  the  air¬ 
craft  system  organizations.  The  magnitudes  shown  in  this  table  are 
consistent  with  Title  14  Federal  Aviation  Administration  regulations, 
like  Special  Federal  Aviation  Regulation  23.787,  which  dictates  that 
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the  equipment  tie-downs  and  restraints  must  sustain  a  9g  inertial  load 
factor  (Aeronautics  &  Space,  2010). 

The  magnitude  of  the  equivalent  load  is  determined  using  the 
magnitude  of  the  sustained  acceleration  load  that  would  be  exerted 
on  a  tie-down  component,  a  critical  part,  or  a  joint  of  the  equipment  in 
a  physical  test.  For  example,  9g  of  acceleration  introduces  90  pounds 
inertial  load  on  a  device  with  10  pounds  of  weight;  and  the  critical 
areas,  such  as  tie-downs  and  restraints,  must  be  tested  to  verify  they 
are  capable  of  restraining  the  inertial  loads  and  moments. 


TABLE  1.  TEST  G-LEVELS  USED  FOR  ACCELERATION  TESTING 


Forward  (G) 

Aft  (G) 

Up  (G) 

Down  (G) 

Lateral  (G) 

9 

1.5 

4 

8 

4 

The  authors  reviewed  the  definition  of  static  load  and  compared 
it  to  the  definition  of  the  sustained  acceleration  described  in  MIL- 
STD-810G  Method  513.6  (Department  of  Defense,  2009).  Shigley  and 
Mitchell  (1983)  define  a  static  load  as  “a  stationary  force  or  moment 
acting  on  a  member”  and  identified  the  following  attributes: 

•  Unchanging  magnitude 

•  Unchanging  point  or  points  of  application 

•  Unchanging  direction 

The  following  statements  are  also  true  for  an  acceleration  test  per 
the  definition  of  acceleration  found  in  MIL-STD-810G  (Department  of 
Defense,  2009): 

•  The  magnitude  of  acceleration  loading  introduced  to 
the  EUT  is  sustained.  In  other  words,  a  test  is  conducted 
for  a  certain  period  of  time  with  a  constant  acceleration. 

•  Acceleration  loading  is  applied  to  each  axis  indepen¬ 
dently;  therefore,  the  point  of  application  does  not 
change  during  a  test. 

•  Each  direction  of  loading  is  applied  independently;  there¬ 
fore,  the  direction  of  loading  does  not  change. 

Table  2  summarizes  these  definitions  for  identifying  the  attributes 
for  a  test  load  to  use  in  the  ELT  method.  This  comparison  helped 
determine  the  properties  of  the  appropriate  test  loads  used  in  ELT 
methodologies.  For  example,  when  a  carrying  handle  is  the  key 
component  used  to  restrain  an  EUT,  the  handle  becomes  the  primary 
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TABLE  2.  ATTRIBUTES  OF  STATIC  LOADING  VS.  EQUIVALENT 
G-LOAD  FOR  TESTING  PER  MIL-STD-810G 


Attribute 

Static 

Loading 

Acceleration  G-Load  in 
Testing  (MIL-STD-810G, 
Method  513.6) 

Test  Load  to  use  in 
the  ELT  Method 

Load 

Magnitude 

Constant 

Sustained  acceleration 

(Constant  for  a  period  of  time) 

Sustained  loading  for 
a  period  of  time 

Point  or 

Points  of 

Application 

Unchanging 

Tests  are  conducted  at  each 

direction  independently 

Unchanging 

Direction 

Unchanging 

Tests  are  conducted  at  each 

direction  independently 

Unchanging 

area  of  concern  for  the  acceleration  tests.  As  such,  the  handle  may 
be  tested  using  the  ELT  method.  The  test  load  would  be  sustained 
for  a  period  of  time,  such  as  t2-t1  as  shown  in  Figure  2.  The  magni¬ 
tude  of  the  test  load  shown  in  Figure  2  (Ftest),  would  be  equivalent 
to  the  magnitude  of  the  inertial  forces  generated  by  the  acceleration 
g-levels,  as  illustrated  in  Table  1.  The  test  load  would  be  applied  to  a 
specific  component  or  components  previously  identified  as  critical 
areas  during  the  acceleration  test  assessment  meeting. 

The  following  sections  describe  both  the  acceleration  test  and 
evaluation  process  and  ELT  method  using  real  case  studies.  The 
ATB  decided  to  apply  the  test  load  for  6  seconds  of  duration  after 
reviewing  the  military  standards  that  describe  the  static  tests.  For 
example,  MIL-STD-209K  (Department  of  Defense,  2005)  states  the 


FIGURE  2.  CONCEPTUAL  LOAD  CURVE  FOR  EQUIVALENT  LOAD 
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"loads  applied  in  the  vertical,  longitudinal,  and  lateral  directions  shall 
be  applied  statically  and  independently  for  not  less  than  6.0  sec¬ 
onds."  This  duration  became  an  ATB  standard  test  parameter  after 
gaining  concurrence  from  the  various  aircraft  system  organizations. 
It  was  consistently  used  when  the  ELT  method  was  selected  for  an 
acceleration  test. 


Case  Studies 

Table  3  shows  some  of  the  acceleration  test  and  evaluation  proj¬ 
ects  implemented  since  May  2008  using  the  new  test  process,  as  well 
as  the  ELT  method  described  earlier.  This  section  discusses  some  of 
the  selected  projects  in  the  next  subsections.  As  listed  in  Table  3,  the 
ATB  team  applied  the  new  process  to  a  total  of  14  projects  and  suc¬ 
cessfully  used  the  ELT  method  in  eight  of  those  projects. 

Case  Study  No.  1:  ELT  Approach  for  Testing  a  Lightweight  Device  (EUT  No.  1) 

EUT  No.  1  is  a  lightweight  medical  device  mounted  in  a  small 
ruggedized  case.  The  item  weighs  6.23  pounds,  and  all  components 
of  the  system  are  confined  inside  the  case.  In  terms  of  acceleration 
testing,  the  only  key  feature  of  this  equipment  was  its  handle  when 
secured  to  a  patient  litter  as  shown  in  Figure  3.  The  team  decided  to 
employ  a  tensile  test  on  the  handle  and  the  handle  mounting  pins  to 
evaluate  the  EUT  under  the  previously  stated  acceleration  criteria. 

Figure  4  shows  the  ELT  setup  for  a  9g  acceleration  test  under  the 
two  worst-case  loading  scenarios  assuming  that  the  equipment  may 
potentially  slide  out  from  between  the  strap  and  litter.  Figure  5  shows 
the  configuration  of  the  EUT  No.  1  on  a  tensile  tester  for  two  orienta¬ 
tions.  These  are  consistent  with  the  identified  test  orientations  shown 


FIGURE  3.  EUT  NO.  1-TIE-DOWN  ON  A  LITTER 


A  New  Process  for  the  Acceleration  Test  and  Evaluation  of 
Aeromedical  Equipment  for  U.S.  Air  Force  Safe-To-Fly  Certification 


October  2010 


49  5 


49  6 


A  Publication  of  the  Defense  Acquisition  University 


http://www.dau.mil 


TABLE  3.  EXAMPLE  ATB  ACCELERATION  TEST  AND  EVALUATION 
PROJECTS 


Project 

Number 

AE  Equipment 
Description 

Case  Study  in 
this  Article 

Test  Method 

ATB-08-01 

Lightweight  device  housed 
in  a  small,  ruggedized 

container 

EUT  No.  1 

ELT 

ATB-08-02 

Portable  oxygen  system  in 
a  protective  cover 

Not  covered 

ELT 

ATB-08-03 

Patient  litter  for 

accomodating  patients  up 
to  250  pounds 

EUT  No.  3 

FEA  (Product- 
Level  Test) 

ATB-08-04 

Patient  litter  for 

accomodating  oversize 
patients 

EUT  No.  3 

FEA  &  ELT 

ATB-08-05 

Mechanical  structure 

to  attach  AE  devices  to 

patient  litters 

EUT  No.  3 

FEA  (Product- 
Level  Test) 

ATB-09-01 

High-pressure  oxygen 
system  in  a  large, 
ruggedized  container 

EUT  No.  4 

Sled  Testing 

ATB-09-02 

Electrical  cables,  plugs, 

and  converters  in  a 

medium-sized,  ruggedized 

container 

Not  covered 

ELT 

ATB-09-03 

Patient  monitor/ 
defibrillator  with  about 

16  pounds  of  weight 

EUT  No.  2 

ELT 

ATB-09-04 

Neonatal  transport  system 
with  heart  and  lung 
support 

SUT  No.  1 

ELT 

ATB-09-05 

Smaller  neonatal  transport 
system  structure 

SUT  No.  2 

ELT 

ATB-09-06 

Mannequin  with  a  control 
system 

Not  covered 

ELT 

ATB-09-07 

Mannequin  with  a  control 
computer 

Not  covered 

ELT 

ATB-09-08 

Stacking  litter  structure 

Not  covered 

ELT  (in  plan) 

ATB-09-09 

Small,  portable  electrical 
generation  system  in  a 
carrying  case 

Not  covered 

Centrifuge  Test 
(in  plan) 
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FIGURE  4.  EUT  NO.  1-TEST  CONFIGURATION  ON  A  TENSILE 
TESTER 

F  equivalent  F  equivalent 

Connection  to 


(a)  (b) 

Note,  (a)  =  Orientation  1;  (b)  =  Orientation  2. 


FIGURE  5.  EUT  NO.  1-ACTUAL  TEST  CONFIGURATIONS 


Note.  The  image  on  the  left  coincides  with  Figure  4  (a),  Orientation  1;  the  image  on  the 
right  coincides  with  Figure  4  (b).  Orientation  2. 
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FIGURE  6.  EUT  NO.  1-LOAD  CURVE  MEASURED  DURING  TENSILE 
TESTING,  ORIENTATION  NO.  2 

6  seconds 


1 


Tensile  extension  (in) 


in  Figure  4.  Figure  6  shows  the  actual  record  of  the  load  applied  for  the 
Orientation  No.  2.  The  test  load  was  held  at  approximately  56  pounds- 
force  for  a  6-second  duration. 

When  considering  a  physical  test  for  EUT  No.  1,  the  ATB  estimated  a 
substantially  higher  cost  for  the  use  of  appropriate  facilities,  fixtures  to 
hold  the  device  during  testing,  and  any  expendable  materials  used  dur¬ 
ing  the  test.  In  addition,  a  physical  test  would  have  required  additional 
planning  and  coordination  time,  thus  driving  schedule  delays  and  add¬ 
ing  additional  costs  for  manpower  spent  during  planning.  In  this  case, 
much  of  these  cost  and  schedule  risks  were  mitigated  by  using  in-house 
tensile  test  stands. 

Case  Study  No.  2:  ELT  Approach  Used  in  Testing  of  a  Portable  Monitor  (EUT  No.  2) 

EUT  No.  2  was  a  lightweight,  portable,  patient  monitor/defibrillator 
weighing  16.2  pounds.  The  team’s  main  priority  was  to  verify  the  item 
could  be  properly  restrained  such  that  it  would  not  become  a  projectile 
during  an  acceleration  event.  To  do  so,  the  team  came  up  with  a  tie-down 
method  using  litter  straps  to  restrain  the  EUT's  movement  in  all  direc¬ 
tions.  If  successful,  this  tie-down  method  would  become  the  approved 
method  for  restraining  the  device  in  the  aircraft  during  operational  mis¬ 
sions.  As  shown  in  Figure  7,  the  litter  strap  passes  through  the  handle  of 
the  EUT  and  the  stirrup  of  the  litter. 
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FIGURE  7.  EUT  NO.  2-TIE-DOWN  ON  A  LITTER 


FIGURE  8.  EUT  NO.  2-TEST  SETUP 


The  team  also  noted  that  when  the  EUT  is  exposed  to  forward 
acceleration  loads,  it  may  potentially  slide  out  from  between  the 
strap  and  litter.  Under  this  scenario,  the  handle  of  the  EUT  would  be 
required  to  bear  the  full  9g  inertial  load  of  the  device.  Therefore,  the 
team  conducted  a  pull  test  to  verify  this  configuration  restrains  the 
EUT.  This  test  would  also  verify  the  ultimate  stresses  of  the  handle 
were  not  exceeded  if  146  pounds  of  equivalent  load  corresponding 
to  9g  inertial  load  factor  were  applied  through  the  EUT’s  CG  in  the 
forward  direction. 

The  team  conducted  a  pull  test  in  the  configuration  shown  in 
Figure  8  using  a  calibrated  force  gauge  for  a  6-second  duration.  By 
using  this  in-house  test  method,  the  team  was  able  to  properly  test 
the  EUT  under  its  operation  configuration  in  less  than  1  hour.  Further, 
this  test  method  only  required  the  purchase  of  a  new  force  gauge  and 
accessories  totaling  $1,250.  If  an  FEA  or  sled  tests  were  used  on  this 
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FIGURE  9.  AE  PATIENT  TRANSPORT  SYSTEM 


device,  estimated  costs  would  start  around  $30,000  and  would  take 
several  weeks  to  plan  and  conduct  the  test. 

Case  Study  No.  3:  FEA  Used  in  Testing  of  Patient  Litters  (EUT  No.  3) 

The  team  recently  evaluated  three  AE  articles  used  to  move 
patients:  two  patient  litters  and  a  special  structure  used  to  attach 
medical  devices  onto  the  litter  during  transport.  In  this  case,  the 
team  decided  to  perform  an  FEA  using  the  ALGOR  static  stress  and 
mechanical  event  simulator  packages.  The  FEA  results  successfully 
identified  components  that  may  fail  under  the  required  acceleration 
loading  on  all  three  AE  equipment  items.  Despite  the  successful  use 
of  FEA  to  identify  potential  safety  risks,  the  time  and  money  spent 
to  evaluate  these  three  devices  were  substantially  higher  than  the 
methods  discussed  in  the  first  two  case  studies.  More  than  $200,000 
was  spent  on  the  FEA  analyses  over  a  one-and-a-half-year  period. 
The  decision  to  conduct  an  FEA  on  these  three  items  was  made  prior 
to  the  development  of  the  ELT  method. 

Case  Study  No.  4:  ELT  Approach  Used  for  Test  and  Evaluation  of  a  Complex 
Transport  System  Structure  ( System  Under  Test  [SUT]  No.  1) 

The  ATB  team  evaluated  a  structure  of  a  complex  system  used 
to  transport  neonatal  and  pediatric  patients  in  critical  condition. 
The  SUT,  shown  in  Figure  9,  contains  13  medical  devices  and  weighs 
about  820  pounds.  Due  to  the  system’s  one-of-a-kind  nature,  as  well 
as  the  cost  and  lead  time  associated  with  procuring  the  advanced 
medical  devices  mounted  within  it,  the  team  consulted  with  several 
of  the  aircraft  systems  organizations  and  decided  on  using  the  ELT 
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TABLE  4.  SAMPLE  TASKS  IDENTIFIED  FOR  THE  ACCELERATION  TEST  AND 
EVALUATION  OF  SUT  NO.  1 


Task  ID 

Component 

Task 

Method 

ATB-SUT1-001 

D-rings 

Perform  a  load  test  to 

verify  that  each  tie-down 
ring  is  capable  of  holding 

1  g  of  acceleration  load. 

Hang  Test  for  D-rings; 

duration:  6  seconds 

Load  analysis  using  the 

12  cargo  straps. 

ATB-  SUn-002 

Casters 

Determine  the  load 

capacity  of  the  casters. 
Identify  the  maximum 
payload.  Determine 
the  load  and  pressure 

distributions  on  the 

aircraft  floor.  Determine 

shoring  requirements. 

Manufacturing  data 

CG  calculation  and 

analysis  for  finding 

the  reaction  loads  and 

pressures. 

Finite  Element  Analysis 
(FEA)  of  the  columns 
and  top  plate  members. 

ATB-  SUn-003 

Locking  Rod 

Perform  structural 

analysis  on  the  locking 

rod  and  side  bracket 

mounting  screws  used  to 
secure  the  locking  rod. 

Finite  Element  Analysis 

ATB-  SUn-004 

Locking  Rod 

Verify  proper  alignment  of 
the  UPS  units  locking  rod 

side  brackets. 

Instruction/Inspection 

ATB-  SUn-005 

Writing  Table 

Include  requirement 
that  the  writing  table  is 
locked/stowed  during 
takeoff,  landing,  and 
emergency  situations. 

Instruction/Limitation 

ATB-  SUn-006 

Padding 

Ensure  foam  padding 

installed  underneath  the 

compressed  gas  cylinders. 

Instruction/Inspection 

ATB-  SUn-007 

Straps 

Verify  the  use  of  straps 

to  restrain  the  handle  of 

Medical  Device  No.  3  to 

the  back  housing  bracket. 

Analyze  strap  strength 
Instruction/Inspection 

ATB-  SUn-008 

Sliding  Shelf 

Assemblies 

Perform  a  pull  test  on  all 
sliding  shelf  assemblies  to 
demonstrate  their  ability 
to  restrain  the  designated 
device  under  4g  lateral 

acceleration  force. 

Pull  tests,  4g,  each 
sliding  shelf  assembly; 

duration:  6  seconds 

Note.  CG  =  Center  of  Gravity ;  UPS  =  Uninterruptible  Power  Supply 
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FIGURE  10.  VIEW  OF  SUT  NO.  1  DURING  HANG  TEST 


FIGURE  11.  WORST  STRESS  FOR  SUT  NO.  1  LOCKING  ROD 


FEA  of  Circular  Beam 
Aeromedical  Test  Branch  (ATB) 


Load  Case  1  of  1 

Maximum  Value:  13211.1  lbf/(in“2) 
Minimum  Value:  528.458  lbf/(in"2) 


Worst  Stress 
lbf/Cin'2) 

Ir  13,211.4400 
-  11,802.2200 
-10,393.0000 
-  8,983.7810 
-  7,574.5610 

-  6,165.3400 
-  4,756.1200 
-  3,346.8990 
-  1,937.6780 
L  528.4578 
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method.  Employing  this  method,  the  team  used  a  combination  of 
analysis,  inspection,  and  supporting  tests  to  satisfy  the  acceleration 
test  requirements.  Therefore,  this  case  study  is  covered  in  more  detail 
than  the  previous  case  studies. 

The  acceleration  test  and  evaluation  team  completed  an  in-depth 
structural  analysis  on  the  SUT  during  which  23  tasks  were  identified. 
These  tasks  included  the  test,  inspection,  and  analysis  of  the  struc¬ 
tural  members,  restraint  mechanisms,  and  tie-downs  of  SUT  No.  1. 
For  a  description  of  the  process,  a  sample  of  eight  of  the  tasks  is 
provided  in  Table  4. 

The  acceleration  test  team  successfully  conducted  tests  and 
performed  analysis  for  each  of  the  tasks  identified  in  the  assess¬ 
ment  meeting.  For  example,  the  eight  D-rings  and  four  additional 
structural  members  were  hang-tested,  and  the  team  subsequently 
verified  that  each  tie-down  location  was  capable  of  withstanding  1g 
acceleration  load.  Figure  10  shows  a  view  of  the  SUT  captured  dur¬ 
ing  one  of  the  hang  tests.  When  each  tie-down  location  is  used,  the 
restraint  capability  for  the  system  can  sustain  at  least  a  9g  forward 
acceleration  load. 

Figure  11  shows  the  analysis  results  for  the  FEA  of  the  locking  rod, 
which  demonstrated  the  locking  mechanism  is  able  to  sustain  1.5g 
lateral  loading.  Additionally,  the  shoring  was  recommended  based  on 
the  FEA  results  and  in  conjunction  with  sample  calculations  provided 
in  MIL-STD-1791(2)  (Department  of  Defense,  1997). 

It  is  important  to  note  that  the  ATB  team  previously  considered 
using  FEA  for  the  acceleration  testing  of  this  SUT.  However,  pro¬ 
jected  cost  and  schedule  figures  similar  to  those  noted  in  Case  Study 
No.  3  discussed  earlier  negated  the  use  of  FEA  on  this  system  and 
required  a  new  approach  to  meeting  the  acceleration  requirement. 
In  fact,  initial  estimates  for  the  FEA  started  around  $474,0  0  0  and 
were  scheduled  to  take  an  estimated  2  years  to  complete.  The  ATB 
is  currently  planning  to  apply  the  same  approach  for  the  test  and 
evaluation  of  two  similar  transport  structures,  saving  an  estimated 
$169,000  and  2  years  of  analysis  time. 

Case  Study  No.  5:  Sled  Testing  of  a  High-Pressure  Oxygen  System  in  a  Large, 
Ruggedized  Container  (EUT  No.  4) 

This  EUT  was  a  bulky,  high-pressure  mechanical  system  used  to 
store  large  volumes  of  medical  grade  oxygen  for  patient  use  during 
transport.  Weighing  nearly  200  pounds,  the  system  is  housed  in  a 
ruggedized  container  and  contains  two  large  compressed  gas  cylin¬ 
ders.  As  mentioned  earlier,  applying  an  ELT  test  method  would  save 
substantial  time  and  money;  however,  using  this  method  would  not 
adequately  test  the  interaction  of  all  components  within  the  system. 
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More  specifically,  the  ELT  method  would  not  test  the  reaction  of  the 
cylinders  to  the  imposed  acceleration  load  and  how  that  reaction 
could  affect  the  gauges,  valves,  and  associated  plumbing.  Based  on 
this  rationale,  the  ATB  team  determined  the  accuracy  and  validity  of 
the  test  data  generated  from  a  physical  test  far  outweighed  its  cost 
and  schedule  risks,  and  the  team  began  planning  a  sled  test  for  the 
EUT.  To  further  improve  the  relevance  of  the  test  and  most  accurately 
mirror  its  operational  configuration,  the  team  elected  to  test  the  EUT 
in  its  pressurized  state,  thus  requiring  the  tests  to  be  conducted  at 
an  outdoor  facility. 

The  selection  of  this  test  method  proved  successful  in  assess¬ 
ing  the  safety  of  this  EUT.  Although  the  tests  cost  roughly  $30,000, 
required  the  construction  of  a  containment  structure  and  two  special 
fixtures  to  hold  the  device,  and  took  over  5  months  to  complete,  the 
data  generated  from  this  sled  test  presented  a  very  detailed  prediction 
of  how  this  EUT  would  perform  during  and  after  an  acceleration  event. 
This  test  also  confirmed  the  intended  tie-down  configuration  was 
capable  of  restraining  the  EUT  during  the  g-loads  shown  in  Table  1. 


Conclusions 

In  the  ever-changing  world  of  acquisitions  and  the  increasingly 
limited  amount  of  money  and  time  for  testing  activities,  the  ATB 
began  exploring  new  test  methods  to  satisfy  acceleration  testing 
requirements  on  AE  equipment.  The  structured  process  described  in 
this  article  continues  to  provide  a  methodical  procedure  for  evaluat¬ 
ing  the  safety  of  medical  devices  and  determining  the  best  method 
or  combination  of  methods  for  conducting  acceleration  tests. 

While  this  article  discusses  only  certain  aspects,  much  of  this 
testing  process  is  founded  on  a  wealth  of  operational  and  technical 
experience.  Additionally,  each  test  article  features  unique  character¬ 
istics  that  do  not  allow  for  standardization  in  the  decision  process. 
Because  of  these  factors,  the  ATB  team  depends  on  the  integrated 
team  construct  to  help  balance  the  decision  process  for  each  project. 

Applying  this  process  has  already  saved  the  ATB  over  $900,000 
in  testing  and  analysis  and  cut  more  than  4  years  from  its  busy  test 
schedule.  This  process,  including  its  dependence  on  an  integrated 
team  approach,  has  the  ATB  poised  to  continue  to  meet  the  demands 
of  the  constantly  evolving  acquisition  environment  in  which  today's 
acquisition  practitioners  must  execute  their  programs. 
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THE  P-51  MUSTANG: 

A  CASE  STUDY  IN 
DEFENSE  ACQUISITION 

i  Alan  Haggerty  and  Roy  Wood 

In  the  rapidly  changing  global  situation,  defense  acquisition 
needs  to  be  equally  agile  and  innovative.  We  must  look  to 
every  source— government,  industry,  and  academia— for  ideas 
to  make  warfighter  systems  more  capable  and  affordable.  This 
article  presents  a  historical  case  study  of  the  World  War  II  P-57 
Mustang  fighter  plane  development  that  illustrates  ways  the 
aircraft  designers  embraced  the  challenge  to  build  a  world-class 
fighter  aircraft  in  the  face  of  a  challenging  enemy,  entrenched 
bureaucracy,  and  immature  industrial  capability.  Enduring 
lessons  are  presented  for  today’s  acquisition  professional. 


Keywords:  P-5 7  Mustang,  P-40  Warhawk,  North 
American  Aviation,  Curtiss-Wright  Corporation,  Risk, 
Aircraft  Manufacturers 
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The  world  is  an  unpredictable  and  dangerous  place.  In  1919,  a 
group  of  men  gathered  to  sign  an  armistice  to  close  the  "war  to  end 
all  wars.”  A  short  two  decades  later,  the  world  was  fighting  an  even 
broader  and  bloodier  war.  In  the  mid-1970s,  Iran,  a  staunch  U.S.  ally, 
was  the  strongest  regional  power  in  the  Gulf.  Five  years  later,  the  Shah 
had  been  deposed,  Iran  was  in  the  midst  of  an  anti-western  revolu¬ 
tion,  and  U.S.  citizens  were  being  held  hostage.  In  1989,  the  Berlin 
Wall  separated  East  from  West,  and  at  Dick  Cheney’s  confirmation 
as  Secretary  of  Defense,  no  one  even  mentioned  Iraq  ("Background 
Briefing,”  2001).  One  year  later,  the  Berlin  Wall  was  down  and  Ameri¬ 
can  forces  were  toe-to-toe  with  an  800,000-man  Iraqi  army.  On 
September  10,  2001,  the  greatest  national  concern  was  the  health  of 
the  stock  market  and  which  dot.com  company  would  be  the  next  to 
go  under.  On  September  11,  everything  changed.  Global  instability, 
nuclear  proliferation,  and  ongoing  armed  conflicts  around  the  world 
threaten  U.S.  security.  More  ominously,  such  instability  and  global 
warfare  appear  to  be  growing  at  an  alarming  pace. 


Defense  Acquisition  and  the 
Changing  Environment 

Yet,  defense  acquisition  appears  to  be  ill  prepared  to  respond  to 
many  of  the  rapidly  emerging  challenges.  As  a  poignant  example, 
improvised  explosive  devices  were  killing  soldiers  and  Marines  in 
Iraq,  but  the  solution— a  heavily  armored  Mine  Resistant  Ambush  Pro¬ 
tected  vehicle— remained  bogged  down  in  a  peacetime  acquisition 
system  until  heroic,  high-level  efforts  broke  through  the  bureaucratic 
obstacles  (DeCamp,  2007;  Feickert,  2008).  On  April  6,  2009,  Secre¬ 
tary  of  Defense  Robert  Gates  refused  to  buy  additional  lots  of  F-22 
fighter  jets,  designed  for  the  Cold  War  and  stuck  in  a  20-year  devel- 
opment-to-delivery  cycle  (Gates,  2009).  At  some  point,  the  question 
arises  as  to  whether  our  defense  acquisition  process  can  ever  be 
as  responsive  as  necessary  to  rapidly  changing  global  threats,  or 
whether,  after  all  is  said  and  done,  its  application  to  today’s  acquisi¬ 
tion  environment  is  largely  irrelevant  and  perhaps  itself  a  danger. 

Has  defense  acquisition  always  been  this  problematic,  or  are 
we  in  a  particularly  difficult  transition  period?  Certainly,  defense 
acquisition  has  always  been  hard.  The  first  ship  procurements  for 
the  U.S.  Navy  in  the  1790s  experienced  cost  and  schedule  overruns, 
congressional  lobbying,  and  technology  overreach  (Toll,  2006).  But, 
the  nation  rebounded  to  produce  resounding  achievements  such  as 
the  nuclear  powered  submarine,  the  intercontinental  ballistic  missile, 
and,  of  course,  the  Manhattan  Project.  These  programs  all  were  begun 
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in  response  to  significant  global  changes  and  dangerous  emerging 
threats.  Defense  acquisition  has  shown  tremendous  responsiveness, 
when  the  need  arises,  to  provide  game-changing  innovations  that 
transform  the  trajectory  of  warfare. 

The  Case  Study  Approach 

What  enduring  lessons,  then,  can  we  learn  from  events  in  history 
that  required  the  system  to  respond  quickly  and  effectively  to  deliver 
these  transformational  systems?  This  article  develops  one  case  study 
to  explore  the  acquisition  challenges  brought  about  by  severe  envi¬ 
ronmental  changes  and  synthesizes  lessons  from  the  case  that  could 
have  applicability  to  our  current  acquisition  system.  This  is  but  one 
case,  albeit  an  interesting  one,  and  risk  is  always  inherent  in  general¬ 
izing  findings  (Yin,  2003).  Nevertheless,  in  the  acquisition  business 
scholar-practitioners  can  gain  valuable  and  perhaps  far-reaching 
insights  through  studying  successful  developments  and  attempting 
to  draw  lessons  from  them;  comparing  and  contrasting  best  practices; 
and  discovering  better  ways  of  plying  the  trade  today. 

Case  of  the  P-51  Mustang 

The  P-51  Mustang  remains  a  highly  recognizable,  legendary  World 
War  II  fighter  aircraft  that  was  the  pride  of  both  the  United  Kingdom 
and  the  United  States.  Development  of  this  innovative  aircraft  was 
fraught  with  challenges— technical,  political,  and  programmatic.  The 
development  story  takes  place  in  a  world  on  the  brink  of  a  second 
global  war,  where  allied  forces  were  largely  unprepared  to  face  an 
enemy  with  better  technology,  war  tactics,  and  wartime  organization. 
In  the  United  States,  neither  the  military,  having  retreated  to  a  peace¬ 
time  pace  after  World  War  I,  nor  industry,  recovering  from  the  Great 
Depression,  was  prepared  for  another  butter-to-guns  transformation 
(Baumol,  Nelson,  &  Wolff,  1994). 

The  year  was  1940.  World  War  II  was  raging,  and  Europe  was 
in  a  desperate  situation.  In  the  United  States,  memories  of  the  hor¬ 
rors  of  World  War  I  were  still  fresh,  and  public  sentiment  had  turned 
decidedly  isolationist.  Despite  early  warnings  from  many  experts, 
the  United  States  and  its  military  were  not  well  prepared  for  another 
global  conflagration. 

With  the  blitzkrieg,  Germany  had  quickly  rolled  up  much  of  conti¬ 
nental  Europe  under  the  Nazi  flag.  The  Battle  of  Britain  was  at  hand, 
and  the  Royal  Air  Force  (RAF)  was  critically  short  of  fighter  aircraft 
to  respond  to  the  coming  German  onslaught.  British  industry  was 
clearly  unable  to  meet  the  RAF’s  production  needs,  so  a  Purchas¬ 
ing  Commission  was  sent  to  the  United  States  in  the  hope  of  finding 
long-range  fighter  aircraft  suppliers  for  its  bomber  escort  missions. 
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The  RAF  agents  initially  approached  the  dominant  U.S.  aircraft 
supplier,  the  Curtiss-Wright  Corporation,  with  a  request  to  place  an 
order  for  more  than  300  of  their  best  fighters— the  P-40  Warhawk— 
which  was  also  the  main  fighter  in  U.S.  Army  Air  Corps  service. 
Curtiss-Wright  turned  the  order  down  due  to  lack  of  factory  capacity. 

The  desperate  British  then  turned  to  a  small  California  com¬ 
pany,  North  American  Aviation,  which  specialized  in  building  training 
aircraft.  The  British  asked  North  American  to  consider  a  licensed 
production  deal  with  Curtiss  to  build  the  Warhawk  in  their  factory. 
The  company’s  president,  “Dutch”  Kindelberger,  asked  for  time  to 
consider  the  offer.  He  knew  that  the  P-40  Warhawk  was  a  relatively 
old  design  that  was  tough  and  heavily  armed,  but  slow  and  lacking 
the  maneuverability  and  combat  performance  to  go  against  the  Ger¬ 
man  Luftwaffe  in  air-to-air  combat. 

After  some  discussion,  the  young  company  president  and  his 
small  design  staff  made  an  astonishing  counter-proposal  to  the 
British.  They  offered  to  design  and  deliver  a  new  airplane,  using  the 
latest  in  aviation  technology.  In  doing  so,  they  promised  the  British  a 
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fighter  of  far  greater  capability,  while  at  the  same  time  leapfrogging 
Curtiss-Wright  and  establishing  their  tiny  company  at  the  forefront 
of  the  international  aviation  industry. 

The  desperate  British  were  taken  aback.  North  American  had 
never  designed  a  combat  aircraft  before.  They  would  have  to  build 
an  entirely  new  factory  and  invent  new  processes  to  manufacture 
the  airplane.  And  to  fulfill  the  contract,  they  would  have  to  produce  a 
world-class  fighter  that  could  outperform  the  German  Air  Force.  Their 
company’s  future,  not  to  mention  the  survival  of  the  United  Kingdom 
and  possibly  that  of  the  entire  free  world,  was  riding  on  their  abilities. 

Amazingly,  the  British  agreed,  with  two  provisos.  First,  North 
American  should  use  the  same  engine  as  the  P-40— the  American 
Allison  V-12.  This  engine  had  a  simple,  one-stage  supercharger  rated 
for  low-altitude  flight,  since  American  doctrine  at  the  time  called  for 
fighters  to  operate  in  direct  support  of  ground  troops  at  low  level. 
Second,  they  had  to  design  and  produce  the  first  prototype  in  less 
than  120  days!  The  company  agreed  to  the  British  conditions  and  went 
to  work  on  what  would  become  the  NA-73  Mustang. 

North  American’s  goal  was  to  build  the  fastest  aircraft  they  could 
make  given  the  limits  of  the  Allison  engine.  Their  designers  decided 
to  use  two  cutting-edge  technologies  that  had  never  been  included 
in  a  production  fighter  aircraft  before. 

The  first  was  the  laminar-flow  wing.  The  laminar-flow  airfoil  was 
the  product  of  massive  investments  in  the  1920s  and  1930s  by  the 
U.S.  government,  specifically  the  National  Advisory  Committee  on 
Aeronautics  (NACA),  the  forerunner  of  the  National  Aeronautics  and 
Space  Administration.  This  design  "smoothed”  the  otherwise  turbu¬ 
lent  airflow  across  the  wing  surfaces,  reduced  drag,  and  increased 
aircraft  speed  and  efficiency  ("Laminer  Flow  Airfoil,”  2010). 

The  second  was  an  untried  cooling  radiator  design  called  the 
Meredith  effect  duct.  The  Meredith  duct  was  essentially  a  divergent- 
convergent  duct  with  a  radiator  at  its  widest  part.  The  theory  was 
that  the  engine’s  waste  heat  would  accelerate  the  flow  of  air  through 
the  duct,  producing  a  ramjet  effect  to  reduce  engine  cooling  drag  at 
high  speeds.  The  design  had  never  been  used  before,  and,  in  fact,  had 
only  been  proposed  as  a  theoretical  possibility  in  an  academic  paper 
in  Britain  between  the  World  Wars  (Meredith,  1936). 

Exactly  117  days  later,  on  borrowed  wheels,  the  prototype  Mustang 
rolled  out  of  the  North  American  factory  (Bowman  &  Laurier,  2007,  p. 
7).  The  British  immediately  placed  a  large  production  order,  and  the 
RAF  Mustang  was  soon  in  front-line  service  as  a  low-altitude  attack 
and  close-support  fighter. 

A  few  months  later,  the  United  States  entered  the  war.  The  U.S. 
Army  Air  Corps  had  several  new  fighters  coming  online,  but  they 
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lacked  a  first-rate  ground  attack  and  tactical  reconnaissance  plane; 
therefore,  the  Corps  ordered  the  Mustang  under  the  U.S.  designation 
A-36  Apache.  The  British  Mustang  and  U.S.  A-36  Apache  served  in 
several  theaters  with  great  success,  and  if  the  story  had  ended  there 
it  would  have  been  a  superb  historic  case  study  of  the  technology 
and  defense  industry  communities  working  together  in  a  time  of 
great  need. 

But,  of  course,  the  story  didn’t  end  there.  The  American  tactic 
of  low-altitude  fighter  combat  proved  to  be  flawed— aerial  dog¬ 
fights  were  high-altitude  affairs  in  the  European  theater.  The  Allison 
engine  was  not  up  to  the  task  because  its  supercharger  lost  power 
at  high  altitude.  The  British,  however,  had  developed  the  Rolls  Royce 
Merlin,  with  a  superlative  high-altitude  supercharger.  This  innovative 
two-speed,  two-stage  supercharger  had  been  designed  by  a  young 
Cambridge  University  mathematician,  Stanley  Hooker,  and  it  allowed 
the  Merlin  to  operate  at  high  power  to  altitudes  above  40,000  feet 
(Hooker,  1984). 

The  new  P-51  Mustang,  with  the  improved  Merlin  engine  and 
larger  fuel  tanks,  went  on  to  dominate  the  air  war  over  Europe,  and 
later  the  Pacific.  The  German  Air  Force  commander  said  after  the  war 
that  he  knew  the  war  was  lost  when  he  looked  up  and  saw  Mustangs 
in  the  skies  over  Berlin  (Rickard,  2007).  In  the  opinion  of  many  histo¬ 
rians,  the  P-51  turned  the  tide  of  the  war  in  Western  Europe  and  was 
crucial  in  gaining  the  final  victory  there. 

Case  Analysis 

The  P-51  Mustang  may  have  been  one  of  the  greatest  success 
stories  in  the  history  of  defense  acquisition  and  a  triumph  of  inno¬ 
vative  technology  insertion.  Note  that  the  Mustang  did  not  spring 
wholly  formed  from  a  highly  structured  requirements  generation 
process  or  from  the  pages  of  some  early  edition  of  DoD  Instruction 
5000.02  (DoD,  2008).  The  Mustang  succeeded  because  its  makers 
were  driven  by  a  combination  of  urgent  warfighting  need,  intense 
industry  competition,  and  the  freedom  to  draw  on  the  intellectual 
forces  of  government,  industry,  and  academia  to  help  them  succeed. 

This  story  may  hold  additional  critical  lessons.  First,  consider 
the  source  of  the  technologies  used  to  give  the  Mustang  its  superb 
performance. 


LAMINAR  FLOW  WING 

The  laminar  flow  wing  came  from  a  government  research  and 
development  agency,  funded  by  the  U.S.  Congress.  Experiments  and 
trials  were  conducted  in  massive  wind  tunnels  at  NACA  labs  in  Lang- 
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ley,  Virginia,  and  at  Moffett  Field  in  California.  NACA  was  solving  an 
interesting  physics  problem,  but  the  solution  had  not  made  its  way 
into  any  application  toward  the  war  effort  until  the  North  American 
team  took  the  concept  and  applied  it  to  the  Mustang.  Similarly,  the 
Meredith  Effect  radiator  duct  was  the  product  of  basic  academic 
research  published  in  a  scholarly  journal. 

Lessons.  The  U.S.  government  continues  to  spend  substantial  sums  on 
basic  and  applied  research  at  laboratories  and  universities  across  the 
country  and  with  our  allies.  Harvesting  that  technology  has  been,  and 
continues  to  be,  devilishly  difficult  and  unpredictable.  Historically, 
that  seems  to  be  the  nature  of  innovation.  When  the  right  programs 
in  need  of  technology  solutions  bump  into  the  right  technologists 
who  have  been  working  on  similar  problems,  magic  happens.  Rather 
than  a  stepwise,  rational  process  to  solve  difficult  problems,  this  is  a 
perfect  illustration  of  the  classic  “garbage  can”  model  of  organiza¬ 
tional  problem  solving  (Cohen,  March,  &  Olsen,  1972).  Here,  a  seeker 
with  a  problem  is  rummaging  around  and  happens  to  find  another 
with  a  potential  solution,  but  who  is  unaware  that  the  problem  exists. 

To  facilitate  this  form  of  “accidental”  problem  solving,  there 
needs  to  be  very  proactive  networking  between  programs-of-record 
actively  seeking  solutions,  with  the  technology  community  who  may 
have  solutions  to  problems  of  which  they  are  unaware.  The  aim  must 
be  to  more  intentionally  force  these  innovation  “collisions”  to  happen 
more  frequently.  All  too  often,  promising  solutions  lie  dormant  waiting 
on  someone  to  pick  them  up,  dust  them  off,  and  look  at  using  them 
in  new  and  fresh  ways. 


SUPERCHARGER  CONCEPT 

Stanley  Hooker’s  supercharger  concept  enabled  the  Mustang  to 
outperform  the  German  Luftwaffe  in  high-altitude  dogfights.  The 
technology  was  not  originally  envisioned,  however,  as  a  military 
improvement.  Rather,  it  was  developed,  tested,  and  matured  in  the 
highly  competitive  environment  of  civil  aircraft  racing  competitions. 

Lessons.  Today’s  downsized  defense  industry,  similar  to  that  of  the 
United  States  and  United  Kingdom  between  World  Wars  I  and  II,  lacks 
robust  competition.  Indeed,  without  a  daunting  enemy  threat,  not 
even  a  clear  vision  or  pressing  need  exists  for  innovation  in  defense 
systems.  However,  a  vigorous  commercial  sector  working  in  an  un¬ 
forgiving  global  competitive  environment  continues  to  develop  new 
and  innovative  products,  many  with  defense-application  potential. 
These  need  to  be  identified  and  encouraged.  Too  often,  when  a  po- 
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tential  military  or  dual-use  item  emerges  from  a  commercial  source, 
it  becomes  subject  to  oppressive  import  and  export  restrictions.  This 
practice  has  a  chilling  effect  on  commercial  innovation,  makes  U.S. 
industry  less  competitive  globally,  and  needs  to  be  changed.  The 
Defense  Department  also  needs  to  help  foster  greater  competition 
among  second-  and  third-tier  defense  and  commercial  subcon¬ 
tractors,  as  well  as  buy  and  use  as  much  unaltered  off-the-shelf 
technology  as  possible  from  non-defense  businesses.  To  facilitate 
transitioning  commercial  technology  to  defense  use,  there  must  be 
a  healthy  market  scanning  ability  within  the  DoD  to  identify  those 
promising  products  and  vendors  that  add  competitive  value  to  our 
programs.  As  seen  in  Iraq  and  Afghanistan,  innovative  insurgents 
and  terrorists  will  require  U.S.  defense,  in  order  to  succeed,  to  also 
exercise  the  cutthroat  entrepreneurialism  for  which  Americans  have 
become  famous. 


SUCCESS  OF  THE  UNDERDOG 

Finally,  “Dutch”  Kindelberger,  the  legendary  head  of  North  Ameri¬ 
can  Aviation,  had  the  technical  savvy,  gambler’s  instinct,  and  almost 
insane  confidence  in  his  company  to  go  head-to-head  with  the  best 
aircraft  manufacturers  in  the  world— and  win. 

Lessons.  The  aviation  industry  of  the  1930s  was  in  its  heyday,  similar  to 
this  era’s  dot.com  and  information  technology  industries,  with  allure 
that  attracted  smart,  aggressive  entrepreneurs.  Aviation  pioneers  of 
the  1930s  hired  bright  young  designers  and  engineers,  and  gained 
confidence  by  cutting  their  professional  teeth  in  the  great  commercial 
air  races  of  the  1930s.  Today,  we  need  to  attract  and  reward  similar 
entrepreneurial  risk  taking  in  defense  and  defense-related  commercial 
industry.  Thousands  of  Kindelbergers  are  out  there  anxious  to  change 
the  world.  The  challenge  is  to  find  and  enlist  them  in  the  effort  to 
maintain  the  strongest  military  on  the  globe. 

The  Secret  Sauce 

None  of  these  insights  are  original,  and  many  might  argue  that 
these  lessons  are  already  being  incorporated  into  the  policies  and 
processes  of  defense  acquisition.  If  the  recipe  is  so  simple,  then  why 
are  current  defense  projects  so  fraught  with  challenges?  Why  can’t 
defense  acquisition  seem  to  tackle  a  modern-day  project  like  the 
Mustang  and  be  just  as  successful? 

The  authors  believe  we  can,  but,  over  the  past  40  years,  we  have 
allowed  ourselves  to  grow  accustomed  to  10-year  missile  devel¬ 
opments  and  20-year  fighter  aircraft  acquisitions.  We’ve  built  a 
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risk-averse  bureaucracy  that  favors  innovation-stifling  oversight  and 
rigid,  failure-intolerant  policies  to  responsible  program  risk  taking  and 
a  sense  of  urgency  in  fielding  weapons  systems.  The  current  acquisi¬ 
tion  system  has  become  so  unwieldy  that  any  sense  of  urgency  or 
spark  of  innovation  is  often  lost  or  frustrated. 


Recommendations 

The  way  ahead  will  be  challenging.  We  have  to  move  from  a 
system  that  imposes  sweeping  requirements  to  one  where  simpler 
is  better,  and  good  enough  is,  well,  good  enough.  The  first  Mustang 
wasn’t  a  war-winning  pony,  but  had  sufficient  design  margin  to  be 
adapted  to  evolving  threats  and  changing  operational  assumptions. 
Today’s  systems  should  be  designed  and  managed  this  way  too. 

We  cannot  allow  our  acquisition  system  to  continue  to  be  so  rigid 
and  risk-averse  that  we  lose  the  opportunity  to  adopt  new  technolo¬ 
gies  when  they  come  along.  The  magic  can  happen  if  we  allow  it,  but 
we  must  be  aware  that  technologies  can  emerge  from  unexpected 
places.  These  are  often  the  real  game-changers.  History  is  replete 
with  examples— the  Internet  (originally  a  nuclear  attack-resistant 
government  network),  stealth  (first  proposed  in  a  Russian  academic 
paper),  and  unmanned  aerial  vehicles  (emerging  from  radio-con¬ 
trolled  scale  models  for  hobby  enthusiasts).  We  should  embrace  the 
garbage  can  model  and  begin  actively  networking  challenged  pro¬ 
grams  with  technologists  who  might  have  solutions. 

Finally,  we  need  to  reenergize  the  sense  of  urgency  that  we  should 
be  feeling  as  our  troops  fight  in  a  prolonged  counterinsurgency  war 
in  southwest  Asia.  While  our  national  survival  is  not  yet  imminently 
at  stake  as  Britain’s  was  in  1940,  we  must  work  to  streamline  our 
processes  to  deliver  needed  battlefield  equipment  much  sooner. 
We  cannot  be  satisfied  with  the  current  system,  and  we  can  never 
become  complacent  or  resigned  to  the  status  quo. 

We  should  study  and  learn  from  the  lessons  of  history,  like  the 
Mustang  story.  From  it,  we  can  remind  ourselves  that  Americans  are, 
by  nature,  innovative  and  entrepreneurial.  We  must  restore  our  self- 
confidence  in  our  ability  to  do  remarkable  things,  remain  steadfast 
in  our  resolve  to  improve  our  system,  become  intolerant  of  bureau¬ 
cratic  obstacles  to  innovation,  and  rededicate  ourselves  to  the  task 
of  making  our  nation  safer  for  ourselves  and  our  children.  The  next 
acquisition  success  story  is  out  there  if  we  can  muster  the  courage 
to  succeed. 
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We  welcome  submissions  from  anyone  involved  in  the  defense  acquisition 
process.  Defense  acquisition  is  defined  as  the  conceptualization,  initiation, 
design,  development,  testing,  contracting,  production,  deployment,  logistics 
support,  modification,  and  disposal  of  weapons  and  other  systems,  supplies, 
or  services  needed  by  the  Department  of  Defense  (DoD),  or  intended  for  use 
to  support  military  missions. 

We  encourage  prospective  writers  to  coauthor,  adding  depth  to  manu¬ 
scripts.  It  is  recommended  that  a  mentor  be  selected  who  has  been  previously 
published  or  has  expertise  in  the  manuscript’s  subject. 

Authors  should  be  familiar  with  the  style  and  format  of  previous  Defense 
ARJs  and  adhere  to  the  use  of  endnotes  versus  footnotes,  formatting  of  refer¬ 
ence  lists  or  bibliographies,  and  the  use  of  designated  style  guides.  It  is  also 
the  responsibility  of  the  corresponding  author  to  furnish  a  government  agency/ 
employer  clearance  with  each  submission. 

MANUSCRIPTS 

Manuscripts  should  reflect  research  of  empirically  supported  experience 
in  one  or  more  of  the  areas  of  acquisition  discussed  above.  Research,  lessons 
learned,  or  tutorial  articles  should  not  exceed  4,500  words.  Opinion  articles 
should  be  limited  to  1,500  words. 

Research  articles  are  characterized  by  a  systematic  inquiry  into  a  subject 
to  discover/revise  facts  or  theories.  Tutorials  should  provide  special  instruction 
or  knowledge  relevant  to  an  area  of  defense  acquisition  and  be  of  benefit  to 
the  Defense  Acquisition  Workforce.  Opinion  articles  should  reflect  judgments 
based  on  the  special  knowledge  of  the  expert  and  should  be  based  on  observ¬ 
able  phenomena  and  presented  in  a  factual  manner,  implying  detachment. 
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STYLE 

We  will  require  you  to  submit  your  final  draft  of  the  manuscript,  especially 
citations  (endnotes  instead  of  footnotes),  in  the  format  specified  in  two  specific 
style  manuals.  The  ARJ  follows  the  author  (date)  form  of  citation.  We  expect 
you  to  use  the  Publication  Manual  of  the  American  Psychological  Association 
(6th  Edition)  and  the  Chicago  Manual  of  Style  (15th  Edition). 

Contributors  are  encouraged  to  seek  the  advice  of  a  reference  librarian 
in  completing  citation  of  government  documents  because  standard  formulas 
of  citations  may  provide  incomplete  information  in  reference  to  government 
works.  Helpful  guidance  is  also  available  in  Garner,  D.  L.,  and  Smith,  D.  H.,  1993, 
The  Complete  Guide  to  Citing  Government  Documents:  A  Manual  for  Writers 
and  Librarians  (Rev.  Ed.),  Bethesda,  MD:  Congressional  Information  Service. 


FORMAT 

Pages  should  be  double-spaced  and  organized  in  the  following  order:  title 
page,  abstract  (120  words  or  less),  list  of  keywords,  body,  reference  list,  author’s 
note  (if  any),  and  any  figures  or  tables.  Figures  or  tables  should  not  be  inserted 
(or  embedded,  etc.)  into  the  text,  but  segregated  (one  to  a  page)  following  the 
text.  When  material  is  submitted  electronically,  each  figure  or  table  should  be 
saved  to  a  separate,  exportable  file  (i.e.,  a  readable  EPS  file).  For  additional 
information  on  the  preparation  of  figures  or  tables,  see  CBE  Scientific  Illustra¬ 
tion  Committee,  1988,  Illustrating  Science:  Standards  for  Publication,  Bethesda, 
MD:  Council  of  Biology  Editors.  Restructure  briefing  charts  and  slides  to  look 
similar  to  those  in  previous  issues  of  the  ARJ. 

The  author  (or  corresponding  author  in  cases  of  multiple  authors)  should 
attach  to  the  manuscript  a  signed  cover  letter  that  provides  all  of  the  authors’ 
names,  mailing  and  e-mail  addresses,  as  well  as  telephone  and  fax  numbers. 
The  letter  should  verify  that  the  submission  is  an  original  product  of  the  author; 
that  it  has  not  been  previously  published;  and  that  it  is  not  under  consideration 
by  another  publication.  Details  about  the  manuscript  should  also  be  included  in 
this  letter:  for  example,  title,  word  length,  a  description  of  the  computer  appli¬ 
cation  programs,  and  file  names  used  on  enclosed  CDs,  e-mail  attachments, 
or  other  electronic  media. 


COPYRIGHT 

The  ARJ  is  a  publication  of  the  United  States  Government  and  as  such  is 
not  copyrighted.  Because  the  ARJ  is  posted  as  a  complete  document  on  the 
DAU  homepage,  we  will  not  accept  copyrighted  articles  that  require  special 
posting  requirements  or  restrictions.  If  we  do  publish  your  copyrighted  article, 
we  will  print  only  the  usual  caveats.  The  work  of  federal  employees  undertaken 
as  part  of  their  official  duties  is  not  subject  to  copyright  except  in  rare  cases. 

In  citing  the  work  of  others,  it  is  the  contributor’s  responsibility  to  obtain 
permission  from  a  copyright  holder  if  the  proposed  use  exceeds  the  fair  use 
provisions  of  the  law  (see  U.S.  Government  Printing  Office,  1994,  Circular  92: 
Copyright  Law  of  the  United  States  of  America,  p.  15,  Washington,  DC:  Author). 
Contributors  will  be  required  to  submit  a  copy  of  the  writer’s  permission  to  the 
Managing  Editor  before  publication. 
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POLICY 

We  reserve  the  right  to  decline  any  article  that  fails  to  meet  the  following 
copyright  requirements: 

•  The  author  cannot  obtain  official  permission  to  use  previously  copy¬ 
righted  material  in  the  article. 

•  The  author  will  not  allow  DAU  to  post  the  article  in  our  ARJ  issue  on 
our  Internet  homepage. 

•  The  author  requires  that  usual  copyright  notices  be  posted  with 
the  article. 

•  To  publish  the  article  requires  copyright  payment  by  the  DAU  press. 
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•  Cover  letter 

•  Biographical  sketch  for  each  author 

•  Headshot  for  each  author  should  be  saved  to  a  CDR  disk  as  a  300 
dpi  (dots  per  inch)  or  high-print  quality  JPEG  or  Tiff  file.  Please  note: 
images  from  web,  PowerPoint  or  e-mail  will  not  be  accepted  due  to 
low  image  quality. 

•  One  copy  of  the  typed  manuscript 

These  items  should  be  sent  electronically,  as  appropriately  labeled  files, 
to  Defense  ARJ  Managing  Editor,  Norene  Fagan-Blanch  at:  Norene.Fagan- 
Blanch@dau.mil. 

In  addition  to  electronic  submissions,  a  hard  copy  of  manuscript  and  any 
diskettes  should  be  mailed  to:  Department  of  Defense,  Defense  Acquisition 
University,  Attn:  DAU  Press  ( Defense  ARJ  Managing  Editor),  Suite  3,  9820 
Belvoir  Road,  Fort  Belvoir,  VA  22060-5565. 
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The  Defense  ARJ  is  published  in  quarterly  theme  editions.  Please  consult 
the  DAU  home  page  for  current  themes  being  solicited.  See  print  schedule 
below. 


2011 


Due  Date 

July  1,  2010 
November  1,  2010 
January  3,  2011 
April  1,  2011 


Publication  Date 

January  2011 
April  2011 
July  2011 
October  2011 


In  most  cases,  the  author  will  be  notified  that  the  submission  has  been 
received  within  48  hours  of  its  arrival.  Following  an  initial  review,  submis¬ 
sions  will  be  referred  to  referees  and  for  subsequent  consideration  by  the 
Executive  Editor,  Defense  ARJ. 


Contributors  may  direct  their  questions  to  the  Managing  Editor,  Defense 
ARJ,  at  the  address  shown  above,  or  by  calling  703-805-3801  (fax:  703-805- 
2917),  or  via  the  Internet  at  norene.fagan-blanch@dau.mil.  The  DAU  Home 
Page  can  be  accessed  at:  http://www.dau.mil. 


DEFENSE  ACQUISITION  UNIVERSITY 

WEB  SITE 

http://www.dau.mil 

Your  Online  Access  to  Acquisition  Research,  Consulting, 
Information,  and  Course  Offerings 


DAU  HOME  PAGE 

•  Performance  Learning  Model 

•  Acquire  (Search) 

•  DAU  Online  Resources 

I  NEED  TRAINING 

•  General  Information 

•  Course  Schedules 

•  Apply  for  a  Course 

•  Pre-Course  Information 

•  Login— Distance  Learning  Courses 

•  Browse  Student  Course  Material 

CONTINUOUS  LEARNING 

•  Register  for  Continuous  Learning  Mod¬ 
ules 

•  Browse  Continuous  Learning  Modules 

•  Login— Continuous  Learning  Modules 

•  Conferences  and  Symposiums 


KNOWLEDGE  SHARING 

•  Defense  Acquisition  Portal  (DAP) 

•  Defense  Acquisition  Policy  Center 

•  Acquisition  Community  Connection  (ACC) 

•  Acquisition  Best  Practices  Clearinghouse 
(BPCh) 

•  ACQuipedia 

•  ACQuire  (Search) 

•  Defense  Acquisition  Guidebook  (DAG) 

•  Integrated  Framework  Chart  (IFC) 

•  Program  Managers  e-Tool  Kit 

•  DAU  Virtual  Library 

•  Media  Library 

MISSION  ASSISTANCE 

•  Description  and  Contacts 

•  Consulting 

•  Facilitated  Decision  Making 

•  Rapid  Deployment  Training 

•  Targeted  Training 

•  Ask  a  Professor  (AAP) 

RESEARCH 

•  Acquisition  Community  of  Practice 
(COP) 


Now  you  can  search  the  DAU  Web  site  and  our  online  publications! 


SURVEY 

Please  rate  this  publication  based  on  the  following  scores: 

5  -  Exceptional  4  -  Great  3  -  Good  2  -  Fair  1  -  Poor 

1)  How  would  you  rate  the  overall  publication? _ 

2)  How  would  you  rate  the  design  of  the  publication?  _ 

3)  Please  list  all  that  apply:  _ 


True  False 

a)  This  publication  is  easy  to  read 

b)  This  publication  is  useful  to  my  career 

c)  This  publication  contributes  to  my  job  effectiveness 

d)  I  read  most  of  this  publication 

e)  I  recommend  this  publication  to  others  in  the  acquisition  field 

4)  What  themes  or  topics  would  you  like  to  see  covered  in  future  ARJsl 


5)  What  topics  would  you  like  to  see  get  less  coverage  in  future  ARJsl 


6)  How  can  we  improve  this  publication?  Provide  any  constructive  criticism  to  help  us  to 
improve  this  publication:  _ 


7) 


Please  specify  your  organization: 


NO  POSTAGE 
NECESSARY 

"  IF  MAILED 

IN  THE 

UNITED  STATES 

BUSINESS  REPLY  MAIL  I 

FIRST-CLASS  MAIL  PERMIT  NO,  12  FORT  BELVOIR. VA  _ _ 


POSTAGE  WILL  BE  PAID  BY  ADDRESSEE 


DEFENSE  ACQUISITION  UNIVERSITY 
ATTN:  DAU  PRESS 
9820  BELVOIR  RD  STE  3 
FORT  BELVOIR  VA  22060-9910 


SUBSCRIPTION 

Defense  ARJ  □  Defense  AT&L  □ 

Quantity _  Quantity _ 

New  Subscription 

□  Cancellation 

□  Change  of  Address 

Date  Form  Completed  _ 

Last  Name  _ 

First  Name  _ 

New  Address  _ 


First  initial 
of  last  name 


Old  Address 


Day/Work  Phone 
E-Mail  Address  (Optional) 


Signature  (Required) 


PLEASE  MAIL  OR  FAX  TO:  703-805-2917 
The  Privacy  Act  and  Freedom  of  Information  Act 

If  you  provide  us  your  business  address,  you  may  become  part  of  mailing  lists  we  are  required 
to  provide  to  other  agencies  who  request  the  lists  as  public  information. 

If  you  prefer  not  to  be  part  of  these  lists,  use  your  home  address.  Please  do  not  include  your 
rank,  grade,  service,  or  other  personal  identifiers. 


FIRST-CLASS  MAIL  PERMIT  NO,  12  FORT  BELVOIR,  VA 


POSTAGE  WILL  BE  PAID  BY  ADDRESSEE 


DEFENSE  ACQUISITION  UNIVERSITY 

ATTN:  DAU  PRESS 
9820  BELVOIR  RD  STE  3 
FORT  BELVOIR  VA  22060-9910 


